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Overview 


Low-income adult learners often struggle to finance their college expenses, having to contend with 
competing responsibilities involving work, family, and school. This report presents findings from a 
study of performance-based scholarships at two community colleges in New York City — the 
Borough of Manhattan Community College and Hostos Community College, both part of the City 
University of New York system. Students who were eligible for the program could receive up to 
$2,600 total over two semesters, or up to $3,900 total over two semesters plus a summer semester. 
Payments were contingent upon maintaining a minimum level of enrollment and grades, and were 
made directly to students. 

The program in New York City is one of six being studied as part of the Performance-Based 
Scholarship (PBS) Demonstration. Each program is designed to serve a different target population 
and to test an alternative incentive structure, and all employ a random assignment research design. 
The program in New York City is a test of a scholarship-only program, with no services attached to 
the award. It targets adult students who are in need of developmental education, and aims to leam 
whether these scholarships are an effective way to help students progress academically. In addition, 
the program’s design allows researchers to test whether offering students a summer scholarship as 
well can further improve their academic outcomes. Analysis suggests that the scholarship-only 
program: 

• Encouraged more full-time enrollment during the semesters in which the program 
operated. Full-time enrollment increased 4. 1 percentage points in the first semester and 6.0 per- 
centage points in the second semester of the scholarship program. 

• Did not increase the average number of semesters registered or credits earned over two 
years. While students who were eligible for the scholarship attempted more credits on average, 
this impact was small and was not accompanied by impacts on the number of semesters regis- 
tered or cumulative credits earned. 

• Increased registration and credit accumulation in the summer semester. However, this 
increase did not lead to increased registration or credit accumulation over the two-year follow- 
up period. 

• Had impacts on students at one college but not on students at the other college. There is 
exploratory evidence that the program at one of the colleges had positive impacts on key aca- 
demic outcomes. The analyses suggest that the impacts varied by college and that the college 
environments may have influenced these impacts. 

MDRC will continue to publish findings from each of the six programs in the PBS Demonstration in 
the coming years. The program in New York makes an important contribution to the overall 
demonstration and to the study of performance-based scholarships. 
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Preface 


While low-income college students are most at risk of not persisting in school to earn a certifi- 
cate or degree, adult learners, who may be dealing with numerous competing priorities such as 
work and child-care responsibilities, often struggle the most among these students. The grant 
aid available to adult learners is not sufficient to cover all of their education-related costs, which 
include tuition, books and supplies, transportation, and basic living expenses. Furthermore, 
adult learners who are in need of developmental education have additional barriers to academic 
success, coupled with the extra cost of developmental courses. Performance-based scholarships 
may be an innovative solution for improving the success of such students. These scholarships 
are need-based, are paid in addition to other existing financial aid programs, and are contingent 
on students’ meeting certain academic benchmarks. 

In 2008, MDRC launched the Performance-Based Scholarship Demonstration seeking to 
evaluate whether performance-based scholarships are an effective way to improve retention 
and degree attainment among low-income students — in different geographical locations, 
with different amounts of monies being provided, and over different durations. This report 
provides findings from an evaluation of a scholarship-only program implemented in 2008 
through 2010 at two community colleges in New York City — the Borough of Manhattan 
Community College and Hostos Community College, both part of the City University of 
New York system. An earlier MDRC report showed that the program encouraged full-time 
enrollment and summer attendance over the period in which students were eligible for a 
scholarship. The new analysis reported here extends the follow-up period, but finds little 
evidence that the program increased academic success at both colleges over this longer 
duration. There is evidence that offering the scholarship over the summer term increased 
registration and credit accumulation in that semester, but these effects did not persist into 
other semesters after the summer scholarship eligibility ended. There is exploratory evi- 
dence that the program had impacts on students at one college but not on students at the 
other college, and that, in the former case, the program had positive impacts on important 
markers of academic success, such as registration, full-time enrollment, and credit accumu- 
lation. This difference in impacts may have arisen because the colleges themselves differ, 
the students in the sample differ, and program implementation varied slightly. 

New York City is one of six programs in six states being evaluated in the Performance-Based 
Scholarship Demonstration. Collectively, these reports will build evidence about the value of 
this innovative way to deliver additional financial support. 


Gordon L. Berlin 
President, MDRC 
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Executive Summary 


The passage of the Higher Education Act of 1965, which extended need-based financial 
assistance to the general population for the first time, has improved college access for American 
students, but more work remains to be done to improve college success. According to govern- 
ment statistics, in 2006, about one in six students had earned a degree or certificate three years 
after beginning their postsecondary education at a two-year institution. 1 

Low-income students are particularly at risk of not persisting to complete a certificate 
or degree — often because of competing priorities, financial pressures, and inadequate prepara- 
tion for college. 2 Among low-income students, older students have added barriers to postsec- 
ondary success. Students in their twenties and thirties often have outside additional obligations 
such as work and child-care responsibilities. The federal and state grant aid available to adult 
learners is often not enough to cover education-related costs, such as tuition, books and sup- 
plies, transportation, and basic living expenses. Moreover, adult learners in need of develop- 
mental education have even greater barriers to academic success, not least among them the extra 
cost of developmental courses. 3 

One promising way to overcome some of these barriers is to offer such students a per- 
formance-based scholarship — a need-based grant, contingent on meeting academic bench- 
marks. The scholarships are designed to help put more money in the hands of low-income 
students and to provide an incentive for making academic progress. A prior MDRC study of 
perfonnance-based scholarships as part of Opening Doors Louisiana found that such scholar- 
ships could have large impacts on persistence, grades, and credit accumulation. 4 This report 
presents findings from a random assignment study of perfonnance-based scholarships at two 
colleges in New York City: the Borough of Manhattan Community College (BMCC) and 
Hostos Community College, both part of the City University of New York (CUNY) system. 


'Lutz Berkner and Susan P. Choy, Descriptive Summary > of 2003-04 Beginning Postsecondary Students: 
Three Years Later, NCES 2008-174 (National Center for Education Statistics, Institute of Education Sciences, 
U.S. Department of Education, 2008). 

2 Lisa Matus-Grossman, Susan Gooden, Melissa Wavelet, Melisa Diaz, and Reishma Seupersad, Opening 
Doors: Students’ Perspectives on Juggling Work, Family, and College (New York: MDRC, 2002). 

’Richard Kazis, Abigail Callahan, Chris Davidson, Annie McLeod, Brian Bosworth, Vickie Choitz and 
John Hopps, Adult Learners in Higher Education: Barriers to Success and Strategies to Improve Results, 
Employment and Training Administration Occasional Paper 2007-03 (Washington, DC: U.S. Department of 
Labor, Employment and Training Administration, Office of Policy Development and Research, 2007). The 
terms “remedial education" and “developmental education" are used interchangeably throughout this report. 

4 Lashawn Richburg-Hayes, Thomas Brock, Allen LeBlanc, Christina Paxson, Cecilia Elena Rouse, and 
Lisa Barrow, Rewarding Persistence: Effects of a Performance-Based Scholarship Program for Low-Income 
Parents (New York: MDRC, 2009). 
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The program in New York City is part of MDRC’s national Performance-Based Scholarship 
(PBS) Demonstration, launched in 2008 to evaluate whether such scholarships are an effective 
way to improve academic outcomes among low-income college students. 5 While some of the 
other programs in the PBS Demonstration (as well as the original Opening Doors study in 
Louisiana) made the scholarship contingent on students’ receiving services such as advising or 
tutoring, the study in New York was intended to test a bare-bones, scholarship-only program. 

The PBS Model and Research Sample in New York 

Performance-based scholarships are need-based grants paid conditionally upon meeting certain 
academic benchmarks around enrollment and grades. They are paid directly to students rather 
than to institutions, and they are paid on top of Pell Grants and any state or institutional aid that 
the student is receiving. An important feature is that they are paid to students regardless of 
academic perfonnance in prior terms, concentrating on the current term of enrollment. That is, 
students do not have to have a specific grade point average or grade requirement from high 
school or prior semesters to qualify for the scholarship. 

In New York specifically, the program was a scholarship-only test, with no other 
prompts or support services required of students. Eligible students were between the ages of 22 
and 35, living away from home, eligible for the federal Pell Grant, enrolled in at least six credit 
hours at BMCC or Hostos, and had tested into (and not yet passed) at least one developmental 
education course. The study sample was recruited over three cohorts of intake, at the start of the 
fall 2008, spring 2009, and fall 2009 semesters. Eligible students who agreed to participate in 
the study were then randomly assigned into three research groups: 

1. The PBS program group, eligible for a scholarship of up to $1,300 each se- 
mester for two semesters (up to $2,600 total) 

2. The PBS-plus-summer program group, eligible for the above scholarship and 
a performance-based scholarship of up to $1,300 for one summer term (up to 
$3,900 total) 

3. The control group, ineligible for the performance-based scholarship, but still 
eligible to receive any other financial aid and scholarships for which the stu- 
dents qualified 


5 See Reshma Patel and Lashawn Richburg-Hayes, Performance-Based Scholarships: Emerging Findings 
from a National Demonstration (New York: MDRC, 2012) for an overview of the programs in each state and 
Lashawn Richburg-Hayes, Paulette Cha, Monica Cuevas, Amanda Grossman, Reshma Patel, and Colleen 
Sommo, Paying for College Success: An Introduction to the Performance-Based Scholarship Demonstration 
(New York: MDRC, 2009) for a more detailed description of four of the six sites. 
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By tracking the groups over time, MDRC is able to determine whether the various 
scholarships cause a difference, or impact, in student outcomes. Random assignment ensures 
that the motivation levels and personal characteristics of students in all groups are equivalent at 
the start of the program; hence, any subsequent differences among the groups can be attributed 
to the program with confidence. The three-group design permits the evaluation to examine 
whether a summer scholarship enhances the effects of the program. 

The three research groups were designed to isolate the effect of the scholarship on 
summer attendance, as the study in Louisiana had suggested that summer attendance might be 
an important factor in explaining the program effects. Scholarships were paid to students in 
increments during the semester as they met three key benchmarks: 

• Initial: $200 after registering for six or more credits or equated credits 6 

• Midterm: $450 upon continued midterm enrollment in six or more credits or 
equated credits 

• Final: $650 after receiving a grade of “C” or better (or a “Pass” in develop- 
mental courses) in at least six credits or equated credits 

During the summer, eligible program students (that is, those in the PBS-plus-summer 
group) were paid in two increments: 

• Initial: $200 after registering for six or more credits or equated credits 

• Final: $450 after receiving a grade of “C” or better (or “Pass” in develop- 
mental courses) in three to five credits or equated credits; or, $1,100 after re- 
ceiving a grade of “C” or better (or “Pass” in developmental courses) in six 
or more credits or equated credits 

While the program targeted students in need of developmental education, students did 
not have to be taking their developmental courses in order to be part of the program or to 
receive scholarship dollars. 

The majority of sample members (almost 80 percent) were recruited from BMCC, with 
a small portion from Hostos. BMCC is the only community college in Manhattan and the 
largest in the CUNY system, serving over 20,000 students in the fall 2008 semester alone. 7 
Conversely, Hostos is located in the Bronx, and is about one-fourth the size of BMCC. The 


6 “Equated credits” are awarded in developmental education courses. These credits do not count toward a 
degree or certificate. 

7 See www.bmcc.cuny.edu and www.bmcc.cuny.edu/iresearch/upload/Fall08FactSheet.pdf. 
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members of the study sample are primarily female, and about half had at least one child. 
Around one-third of sample members said they were the first in their family to attend college. 
Compared with members of the sample at BMCC, students in the Hostos sample were more 
likely to be Hispanic, to speak a language other than English at home, and to be receiving 
government benefits. 

Program Implementation 

The design of the program in New York was relatively straightforward, and the program was 
implemented with fidelity to its design. Overall: 

• The two colleges successfully implemented the scholarship as intended. 

Scholarships were paid in multiple installments, and both colleges had designated staff 
to process the payments. Site coordinators examined transcripts for students in the program 
group to determine whether students met the criteria to earn a scholarship. They then took the 
necessary actions to trigger a payment to the student. While the details of the payment process 
differed slightly at the two colleges, in general, staff communicated with program group 
students at every payment point with a letter congratulating them (if applicable) and reminding 
them of their next payment. 

The program at the colleges was not intended to have a counseling component, and 
program staff members were expected to restrict their interactions with students to answering 
questions related to their scholarships. The program at BMCC was housed in the financial aid 
office, and students were not likely to receive any guidance or counseling that was not intended 
to be part of the program. However, at Hostos, the program was housed in the student services 
division. When students visited the program office, they were likely to have access to a number 
of other services including career counseling, social services, academic advising, and the 
financial aid and registrar’s office. 

• Almost all program group members received at least one payment. 

Overall, in the first two semesters of the program, 99.3 percent of program group mem- 
bers received at least one payment. On average, students received $2,002 over the two semes- 
ters (over 75 percent of the dollars available to them). In the summer semester, the PBS-plus- 
summer group was eligible for an additional $1,300 summer scholarship. In this group, over 30 
percent of students received an initial payment, and about one-fourth of eligible students 
received a final payment. The majority of the PBS-plus-summer group did not register for 
summer courses, leaving valuable scholarship dollars on the table. 

• Control group members did not have access to a si mi lar program. 
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At both colleges, financial aid data collected show that program group members were 
awarded more financial aid than control group members. Both groups were awarded similar 
types and amounts of financial aid other than the perfonnance-based scholarship; the key 
difference between their financial aid packages was the perfomiance-based scholarship. 

• In qualitative interviews, students in both groups indicated that as older 
college students, they faced particular challenges. Overall, program 
group members felt that the scholarship did not have a large influence 
on their thinking and behavior. 

Around 20 interviews at both colleges, representing a mix of program and control group 
members, were completed near the end of the program. Students cited balancing work, school, 
and family demands as challenges to their success at school, and most students said that 
financial aid made it possible for them to go to college. Program group students did not believe 
the scholarship affected their performance, but most reported that it helped relieve some of their 
financial burdens. 

Overview of Impact Findings 

This report examines transcript data collected from the two colleges and follows students for 
four main semesters (main semesters are considered fall and spring semesters) and any summer 
or winter sessions in between. Early findings were presented in a report in May 2011, which 
examined the first two cohorts of students to enter the study and showed that the perfonnance- 
based scholarship encouraged full-time enrollment in the first two semesters and increased 
summer attendance. 8 The main findings of this follow-up report are: 

• The scholarship-only program encouraged more full-time enrollment 
during the program semesters. 

Full-time enrollment increased 4. 1 percentage points in the first semester and 6.0 per- 
centage points in the second semester of the scholarship program. These findings are consistent 
with the early findings report from May 2011, though slightly smaller in magnitude. 

• The scholarship-only program did not increase the average number of 
semesters registered or credits earned for the full sample over the two- 
year follow-up period. 


8 Lashawn Richburg-Hayes, Colleen Sommo, and Rashida Welbeck, Promoting Full-Time Attendance 
Among Adults in Community College: Early Impacts from the Performance-Based Scholarship Demonstration 
in New York (New York: MDRC, 2011). 
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While program group students attempted more credits on average, this impact was fair- 
ly small and was not accompanied by impacts on the number of semesters registered. This 
impact also did not translate into impacts on cumulative credits earned. The early impacts from 
the May 20 1 1 report were sustained with the full sample; however, these early impacts do not 
translate into any impacts overall during the follow-up period. This suggests that while the 
program was effective when students were eligible for scholarships, the effects on enrollment 
and credits earned dissipated after the program ended. 

• There is evidence that offering the scholarship over the summer term 
increased registration and credit accumulation in that semester. 

PBS-plus-summer students registered for summer courses at a higher rate, and attempt- 
ed and earned more credits on average. However, there is no evidence that providing a summer 
scholarship during one summer increased the cumulative number of semesters registered, 
credits attempted, or credits earned over the two-year follow-up period. 

• There is exploratory evidence that the scholarship-only program had 
impacts on students at one college but not on students at the other col- 
lege. The program at one of the colleges had positive impacts on im- 
portant markers of academic success. 

The impacts of the scholarship-only program were estimated separately for each col- 
lege. These analyses were not designated ahead of time, and thus are exploratory only; however, 
there are significant impacts at Hostos on registration, full-time enrollment, and credit accumu- 
lation, and these effects persist after the scholarship ends. Figure ES. 1 details the credit accumu- 
lation during the first four semesters after random assignment at Hostos alone. The program 
group earned more cumulative credits in every semester after random assignment, and it is 
notable that these impacts continue to grow in the absence of the scholarship. Overall, during 
the follow-up period, program group students at Hostos earned over six more credits than 
control group students. Similar impacts were not found on any academic measures at BMCC. 

There are a few possible explanations for the differences in the program’s effectiveness 
at the two colleges. First, the colleges themselves are different, as BMCC is about four times the 
size of Hostos. Second, the students in the study sample are different. Hostos students tend to be 
older, more likely to be parents, more likely to be Hispanic, and more likely to be receiving 
government benefits such as Temporary Assistance for Needy Families or food stamps. Last, 
the program at Hostos may have been implemented as somewhat more than a scholarship-only 
program, as it was housed in the student services division where students had more access to 
services such as academic advising. In all, it is possible that the context in which these programs 
operated may have influenced these impacts. 
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The Performance-Based Scholarship Demonstration 
Figure ES.l 

Credit Accumulation During the First Four Semesters, Hostos Community College 

New York Final Report 

60 

50 


40 



First semester Second semester Third semester Fourth semester 


■ Program □ Contro 1 

SOURCE: MDRC calculations from Flostos Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

Analysis is considered exploratory because subgroup was not designated ahead of time. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits are 
not included in the outcomes shown in this figure. 
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• The total cost per student at the two colleges was roughly $3,000. 

This total cost comprised around $2,100 in scholarship costs and $900 in support and 
administration. The scholarship costs represent the amount of money the program paid directly 
to students. Support and administration includes all other costs such as salaries, benefits, 
overhead associated with program staff, supplies, materials, travel, and professional develop- 
ment. 

Conclusions and Next Steps 

While there was some initial promise in the early semesters of the program in New York, these 
impacts did not ultimately lead to an increase in credits earned or the number of semesters 
registered. Similarly, impacts over the summer were initially promising, but did not translate 
into long-tenn academic gains. When the results are examined by college, there is evidence that 
the program worked better at Hostos; however, these results are exploratory in nature and 
should be an area for further research. Last, the total cost per student at the two colleges was 
roughly $3,000 — comprising around $2,100 in scholarship costs and $900 in support and 
administration. 

The study in New York is at one end of the spectrum of perfonnance-based scholar- 
ships — a scholarship-only program with an academic benchmark required to get the scholar- 
ship payment and no advising or tutoring component. As noted earlier, New York is just one 
test in the six programs being studied as part of the PBS Demonstration. Other programs have 
additional components to the intervention, designed to help support students in their studies. 
Additionally, the New York study targets a very specific population — adult learners in need of 
developmental education. The target populations vary across the demonstration, in order to help 
determine which types of students are best suited for the scholarships. MDRC will continue to 
publish findings on each of these programs in the coming years. Collectively, these findings will 
add to the body of knowledge on the effectiveness of these scholarships in improving the 
academic success of low-income students. 
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Chapter 1 

Introduction 


This report presents two-year results from a rigorous evaluation of a perfonnance-based 
scholarship program in New York City, targeting students ages 22 to 35 who are in need of 
developmental education. Perfonnance-based scholarships are need-based grants whose 
disbursement is contingent on students’ meeting academic benchmarks after enrolling in the 
program. The scholarships are designed to help put more money in the hands of low-income 
students and to give students an incentive to change their behavior with respect to their academ- 
ic coursework. In New York City, the program was intended to test a scholarship-only program 
tied to academic benchmarks, without the inclusion of student services such as advising or 
tutoring. 

The program in New York City is part of MDRC’s national Performance-Based Schol- 
arship (PBS) Demonstration, launched in 2008 to evaluate whether such scholarships are an 
effective way to improve academic outcomes among low-income college students. 1 The 
evaluation uses a random assignment design, similar to those used in medical trials, to measure 
the effects of the program on academic markers of success. While other programs in the 
Performance-Based Scholarship Demonstration made the scholarship contingent on students’ 
using certain student services such as advising or tutoring, the study in New York was intended 
to test a bare-bones, scholarship-only program. Students who met the eligibility criteria in New 
York were randomly assigned to one of the following three groups: 

1. The PBS program group, eligible for a scholarship of up to $1,300 each semester 
for two semesters (up to $2,600 total) 

2. The PBS-plus-summer program group, eligible for the above scholarship and a per- 
fonnance-based scholarship of up to $1,300 for one summer tenn (up to $3,900 total) 

3. The control group, ineligible for the performance-based scholarship, but still eligible 
to receive any other financial aid and scholarships for which the students qualified 

By tracking the groups over time, MDRC is able to determine whether the various 
scholarships cause a difference, or impact, in student outcomes. Random assignment ensures 
that the motivation levels and personal characteristics of students in all groups are similar at the 
start of the program; hence, any subsequent differences can be attributed to the program with 

'See Patel and Richburg-Hayes (2012) for an overview of the programs in each state and Richburg-Hayes 
et al. (2009b) for a more detailed description of four of the six sites. 
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confidence. The three-group design permits the evaluation to examine whether a summer 
scholarship enhances the effects of the program. 

In addition to describing the program at the New York colleges, its implementation, 
and detailed academic findings related to it, this report provides background information on the 
national demonstration and how the program in New York adds to a growing body of 
knowledge. Further, the report contains information on the cost of the program and some 
important lessons learned for future research. 

The National Performance-Based Scholarship Demonstration 

As noted earlier, this program, started in fall of 2008, is one of six programs being studied as 
part of MDRC’s national PBS Demonstration. The demonstration is supported by a consortium 
of funders, partners, and postsecondary institutions. MDRC evaluated a similar performance- 
based scholarship program in 2004 as part of its Opening Doors Demonstration, 2 which tested 
several interventions to improve student success at community colleges. Low-income parents at 
two New Orleans community colleges enrolled in a program that allowed them to earn up to 
$1,000 per semester for two semesters (or $2,000 total) provided that they maintained at least 
half-time enrollment and a 2.0 (“C”) grade point average. The program operated during the 
2003-2005 school years. The evaluation found that the scholarships had positive effects on 
several outcomes, including students’ credit accumulation and semester-to-semester retention. 
These effects persisted into the third and fourth semesters, when most students were no longer 
eligible for the scholarship. Just after the program ended, Hurricane Katrina struck the Gulf 
Coast region, causing severe destruction and temporarily shutting down the two colleges. Many 
students in the study moved away, and the devastation inflicted made it virtually impossible to 
determine whether there are long-term effects on graduation and transfer rates. 

While the Louisiana study is an important contribution to the literature, it is just one 
test. The goal of the PBS Demonstration is to build more evidence about whether performance- 
based scholarships help at-risk students succeed academically and persist at higher rates than 
they normally would in the absence of such an intervention. Eight colleges and one intermedi- 
ary organization across six states are participating in the demonstration. While the amount and 
duration of the scholarships, perfonnance criteria, and target groups for the intervention vary 
among the programs, all programs offer an incentive-based scholarship designed to address the 


2 Richburg-Hayes et al. (2009a). 
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needs of low-income students . 3 The intervention in each state is also being evaluated using 
random assignment, similar to the program in New York. 


Key Principles of Performance-Based Scholarships and the 
Theory of Change 

While New York is unique within the larger suite of programs in the PBS Demonstration, in 
that the programs vary by target population, performance benchmarks, scholarship amounts, 
and the integration of student services (among other things), all of the studies incorporate three 
key principles of performance-based scholarships: 

• They are paid contingent on students’ meeting basic conditions regarding en- 
rollment and grades in college courses, thus acting as an incentive to reward 
behavior associated with academic success. Unlike merit-based scholarships, 
the scholarships are paid to students based on their academic performance in 
the current term, regardless of their performance in previous terms. 

• To reinforce the incentive nature of these scholarships, they are paid directly 
to students rather than to institutions and allow students to use the money to 
cover both educational and noneducational expenses (for example, child care 
and transportation). 

• They supplement Pell Grants and state aid to help meet the needs of low- 
income students. In other words, the intervention generally results in stu- 
dents’ receiving more money to cover academic and living expenses, and can 
potentially reduce their dependency on loans . 4 

The theory of change underlying these scholarships is that conditioning additional fi- 
nancial aid on certain behaviors will lead to an increased effort toward studies, a reduced level 
of financial stress, and an increased confidence on the part of students in their ability to suc- 
ceed . 5 These changes are hypothesized to result in an increase in enrollment in subsequent 
tenns, an increase in credits attempted, and an increase in credits earned. Increases in these 
short-term educational outcomes may then lead to changes in medium- and long-term educa- 
tional outcomes, including the total cumulative credits earned, as well as graduation or transfer 


3 See Patel and Richburg-Hayes (2012) for an overview of the early findings across the states in the 
demonstration. 

financial aid regulations prohibit certain students from receiving financial aid in excess of their need. In 
these instances, federal work-study or loans may be displaced by the performance-based scholarship. 

5 These are the pathways that the intervention is theorized to follow. However, the analysis in this report 
does not attempt to validate these mediators. 
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to a four-year institution. If the effects on educational outcomes are positive and strong, the 
intervention could in turn lead to improved labor market outcomes in the longer term, including 
higher earnings. 

In this report, the project seeks to answer two main questions: What is the impact of the 
performance-based scholarship over two years on the key academic outcomes of persistence, 
credit attempts, and credit completions? And, are the effects of the intervention enhanced by 
offering funding for summer attendance? 

The PBS Evaluation in New York City 

The program in New York is a unique study within MDRC’s national PBS Demonstration for a 
number of reasons. First, the program is focused solely on providing scholarships, without 
additional supports or counseling, thus providing a true test of a scholarship-only program. The 
original Opening Doors study in Louisiana had an additional counseling component attached to 
the scholarship dollars, as do three of the six programs being studied as part of the broader PBS 
Demonstration. Thus, the New York program is the “bare bones” version of the performance- 
based scholarship program — it is testing whether additional dollars, contingent on an academic 
benchmark, with no counseling, tutoring, or other support requirements, can improve the 
academic outcomes for a low-income population. 

Additionally, the program targets students in need of at least one level of developmental 
education who may be particularly at risk of not having enough funds to cover their college 
expenses. Many students arrive on college campuses unprepared to do college-level work — in 
fact, 42 percent of first-year students at community colleges enroll in at least one developmental 
reading, writing, or mathematics course. 6 Such courses are designed to bring students’ basic 
skills to college-level standards; however, students in need of developmental courses have 
lower college completion rates overall than other students at community colleges. 7 As well as 
being a test of a scholarship-only program, the program in New York can answer the question 
of whether providing developmental education students with additional financial aid, and no 
other supports, is enough to help them succeed academically. The program is one of two 
programs in the demonstration specifically targeting students in need of some remediation. 

Finally, the New York study is the only three-way design in the demonstration, allow- 
ing an analysis of whether the effects of the intervention are enhanced by offering funding for 
summer attendance. The results of Opening Doors Louisiana suggested that the scholarships 
may go further if offered for the summer term, but the effects of summer attendance could not 

6 Aud et al. (2011). 

7 Attewell, Lavin, Domina, and Levey (2006). 
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be disentangled from the effects of the scholarship offer. The test in New York provides a 
unique opportunity to leam more about the impact of offering scholarship dollars in the summer 
specifically. 8 

The Colleges and Staff 

This study was implemented at Borough of Manhattan Community College (BMCC) 
and Eugenio Maria de Hostos Community College during the fall 2008 through summer 2010 
semesters. The study sample was recruited over three periods of intake, at the start of the fall 
2008, spring 2009, and fall 2009 semesters (for three “cohorts” of students). Almost 80 percent 
of the New York sample attended BMCC, with the remainder attending Hostos. 

Both colleges had designated staff to manage and implement the program — the Direc- 
tor of Financial Aid at BMCC and the Assistant Dean for Student Development and Enrollment 
Management (in conjunction with the Director of Financial Aid) at Hostos. In addition to the 
senior staff, who had overall management responsibilities, both colleges had coordinators who 
were responsible for the day-to-day implementation of the program. 

The Target Population 

As noted earlier, the program in New York City targeted low-income students at 
BMCC and Hostos. The program targeted both new and continuing students, and to be eligible, 
students were required to: 

• Be between the ages of 22 and 35 

• Be living away from their parents (a proxy for independent status) 

• Be eligible for a federal Pell Grant 

• Have tested into (and not yet passed) at least one developmental education 
course 

• Be enrolled in at least six credit or contact hours at study intake 9 

These requirements were specifically chosen to target students who were most at risk of 
not having enough funds to cover the cost of their postsecondary education, for reasons de- 
scribed in detail in Box 1.1. 


8 See Richburg-Hayes, Sommo, and Welbeck (2011) for further information. 

9 There were a handful of cases at Hostos where students in the fall 2009 cohort were not registered in fall 
2009. These cases make up less than 1 percent of the overall sample. 
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Box 1.1 


An Overview of Financial Aid in New York State 

The primary need-based financial aid program for college students in the United States is the 
federal Pell Grant program, which makes awards to students based on the cost of attendance at 
an institution,* the Expected Family Contribution, 1 and the student’s attendance status (that is, 
full time, three-quarter time, half time, or less than half time). In addition to the federal Pell 
Grant, many states provide need-based grants to state residents who attend postsecondary 
institutions within the state. New York State’s Tuition Assistance Program (TAP) is among the 
most generous state-funded financial aid programs. The program requires that students matric- 
ulate full time during their first year in order to be eligible for a full-time or part-time grant in 
subsequent semesters. 

While the Pell Grant program and TAP provide a good foundation of need-based aid for New 
York State residents attaining postsecondary education, adult students in need of developmen- 
tal education often have lower financial aid payments than traditional students. Adult students 
who are independent have an award range of $500 to $3,025, while dependent students (or 
traditional college-age students) have awards of up to $5,000 per year. Additionally, adult 
students are often attending college part time, which usually results in lower grant amounts for 
both the Pell Grant and TAP. While TAP provides six semesters of support for an associate’s 
degree, this eligibility is likely to run out for students who need to meet developmental educa- 
tion requirements before being able to complete a two-year degree. 1 Last, most adult students 
work to support themselves, and these earnings can prevent them from getting the maximum 
amount of TAP funds available. 5 Taking these factors together, low-income, independent, 
adult students attending school part time and in need of developmental education are particu- 
larly in danger of having inadequate funds to cover their postsecondary education costs. For 
that reason, the PBS New York program was specifically designed to target this population. 

NOTES: The total cost of attendance includes tuition and fees, books, transportation, and room and board. 
Financially independent students tend to have higher living expenses (including room and board) than 
traditional students; thus, their total cost of attendance is higher. 

The Expected Family Contribution (EFC) is the amount of money that a family is expected to be able 
to contribute to a student’s education, as calculated according to federal guidelines. 

Milliard (2007). 

Tbior wages are taken into account in the Free Application for Federal Student Aid (FAFSA), and the 
calculation for EFC assumes that the prior income is available for the upcoming academic year. Also, 
changes in employment status require the student to initiate special procedures. Even so, the calculations do 
not fully account for a student’s potential shortage of hinds available for college. See Kazis et al. (2007). 


The Program Model 

All program group students were eligible for the scholarship of up to $1,300 for two 
consecutive semesters (excluding summer). The scholarship was paid directly to students in 
increments as they met three key benchmarks: 
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• Initial: $200 after registering for six or more credits or equated credits 10 

• Midterm: $450 upon continued midterm enrollment in six or more credits or 
equated credits 11 

• Final: $650 after receiving a grade of “C” or better (or a “Pass” in develop- 
mental courses) in at least six credits or equated credits 

The first and second payments are designed to offer an incentive to maintain a mini- 
mum level of attendance, while the last payment is designed to offer an incentive for perfor- 
mance. In order to receive the initial payment, each student had to have paid his or her tuition or 
established a payment plan. As a result, the study’s definition of registration is more stringent 
than the usual definition, since students who have reached the point of paying or establishing a 
payment plan are more likely to attend classes than students who have only registered for 
classes. This also resulted in virtually all program group students receiving the first payment. 

As mentioned earlier, half of the program group was eligible for an additional $1,300 
during the summer semester, for a total amount of up to $3,900 across three semesters. This 
allowed researchers to test the effect of the scholarship on summer attendance. The Opening 
Doors results in Louisiana suggested that summer attendance might be an important factor in 
explaining the program’s effects, and the research design in New York was structured, in part, 
to answer this question more definitively. 12 During the summer, eligible program group students 
were paid if they met the following benchmarks: 13 

• Initial: $200 after registering for three or more credits or equated credits 

• Final: Part-time benchmark: $450 after receiving a grade of “C” or better (or 
a “Pass” in developmental courses) in three to five credits or equated credits 

OR 

• Final: Full-time benchmark: $1,100 after receiving a grade of “C” or better 
(or a “Pass” in developmental courses) in at least six credits or equated credits 


,0 Note that equated credits are awarded in developmental education courses. These credits do not count 
toward a degree or certificate. 

1 1 Students received this payment if they attended at least once in the first three weeks and at least once 
during the fourth or fifth week of the semester. 

12 See Richburg-Hayes, Sommo, and Welbeck (2011) for further information. 

13 Students were also eligible for the summer scholarship if they participated in the colleges’ summer im- 
mersion programs. Due to the complexities of administering the scholarship in the summer, summer immer- 
sion students in the PBS-plus-summer program were not eligible for the summer performance-based scholar- 
ship for the last cohort (fall 2009). 
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While the program targeted students in need of at least one level of developmental edu- 
cation, it did not require students to take their developmental courses in order to be a part of the 
program or receive their award. However, students could count developmental courses toward 
their academic benchmarks and had to pass these courses in order for them to count toward the 
six-credit requirement (as opposed to obtaining a “C” or better in the nondevelopmental 
courses). These criteria were intentional — if developmental courses were not counted toward 
the scholarship requirement, students might have been discouraged from completing their 
remedial coursework in a timely manner. 

Recruitment and Random Assignment 

The process for recruiting sample members and randomly assigning them into the three 
research groups is detailed in Figure 1.1. The process had three general phases: inviting students 
to participate, enrolling students in the program, and randomly assigning them into one of the 
three research groups. The staff at both colleges had a systematic recruitment plan in order to 
reach the sample-size goals for the study. This plan involved sending invitation letters to 
students meeting the eligibility criteria, creating an online reservation system, and conducting 
follow-up phone calls to encourage students to attend an orientation session. 

The enrollment of students began with a presentation followed by a question-and- 
answer period, and ended with interested students completing an informed consent form and an 
online survey. This online survey asked students for demographic and other basic information, 
and is referred to as a Baseline Information Form (BIF). 

After the session was complete, MDRC randomly assigned students to the program 
group that was eligible to receive the scholarship over two semesters (PBS group), the program 
group that was eligible to receive the scholarship over two semesters plus summer (PBS-plus- 
summer group), or to the control group, which was not eligible for the scholarship. College staff 
members were concerned that randomly assigning students at the orientation sessions could 
result in a strong emotional reaction, since some students would be given the opportunity to 
earn the award while others would not. Thus, random assigmnent was conducted after the 
session was complete, and students were mailed letters informing them of the research group to 
which they had been assigned. 

Recruitment for the study spanned three semesters at BMCC (fall 2008, spring 2009, 
and fall 2009), and two semesters at Hostos (spring 2009 and fall 2009). The recruitment effort 
was successful, and both colleges met their sample size goals within the number of semesters 
projected. 



The Performance-Based Scholarship Demonstration 
Figure 1.1 

Basic Random Assignment Process for the Demonstration in New York 

New York Final Report 
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Data Sources 

Several data sources were used for the analyses presented in this report; these are described below. 

• Baseline data: As mentioned above, students completed a baseline ques- 
tionnaire that asked them for demographic and other background infomiation 
before they were randomly assigned to the program group or control group. 
Baseline data are used to describe the sample, to assess the success of ran- 
dom assignment, and for subgroup analyses. 

• Scholarship payment data: MDRC obtained performance-based scholar- 
ship payment records from BMCC and Hostos. Findings on these payment 
data are used to describe program take-up rates. 

• Financial aid data: The central office at The City University of New York 
provided financial aid award data for all sample members in the program 
semesters. These data include information on the performance-based scholar- 
ship, as well as federal Pell Grants, New York State’s Tuition Assistance 
Program, student loans, and any other scholarships and grants administered 
by the colleges. Data are provided for the fall and spring program semesters. 

• Scholarship disbursement visits: MDRC visited both BMCC and Hostos 
staff to detail the process for scholarship payments and to do a quality assur- 
ance check of their scholarship disbursement process. 

• Student interviews: In addition to the administrative data on students, MDRC 
fielded one-on-one interviews with about 20 students (a mix of program and 
control group members from the two colleges) in May and August 2010. 

• Transcript data: Both colleges provided transcript data for the sample mem- 
bers in the study. These data encompass measures like enrollment status, cred- 
its attempted and earned, and grade point averages, and were used to provide a 
detailed look at sample members’ perfonnance in college. Data are provided 
over four main semesters of follow-up (main semesters are considered fall and 
spring semesters), as well as any summer and winter sessions in between. 

Characteristics of the Performance-Based Scholarship Sample 

As described above, students in the study sample completed a baseline questionnaire that asked 
them for demographic and other background infomiation before they were randomly assigned 
to the program or control groups. The first column of Table 1.1 shows characteristics for the full 
sample, and the subsequent columns present this infomiation separately for each college. 
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The Performance-Based Scholarship Demonstration 

Table 1.1 

Selected Characteristics of Sample Members at Baseline, by Site 
New York Final Report 


Characteristic 

Full 

Sample 

BMCC 

Hostos 

Gender (%) 

Male 

30.9 

33.7 

21.0 

Female 

69.1 

66.3 

79.0 

Age (%) 

22-26 years 

57.3 

60.0 

48.0 

27-30 years 

24.3 

23.0 

28.8 

3 1 years and over 

18.4 

17.0 

23.1 

Average age (years) 

26.5 

26.4 

27.1 

Marital status (%) 

Married 

18.2 

16.6 

24.0 

Unmarried 

74.0 

76.2 

66.1 

Missing 

7.8 

7.2 

9.9 

Race/ ethnicity 3 (%) 

Hispanic 

44.3 

37.5 

67.7 

White 

6.1 

7.5 

1.2 

Black 

37.2 

39.5 

29.0 

Asian or Pacific Islander 

9.7 

12.3 

0.9 

Other 

2.7 

3.2 

1.2 

Number of children (%) 

0 

52.3 

55.4 

41.3 

1 

26.5 

26.2 

27.7 

2 

14.4 

12.2 

21.9 

3 or more 

6.8 

6.2 

9.1 

Among sample members with children 

Average age of youngest child (years) 

4.3 

4.2 

4.6 

Household receiving any government benefits (%) 

37.9 

34.8 

48.6 

Missing 

12.0 

12.6 

9.9 

Financially dependent on parents (%) 

1.3 

1.1 

1.8 

Missing 

4.4 

3.9 

6.0 

Currently employed (%) 

56.0 

55.9 

56.3 


(continued) 
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Table 1.1 (continued) 


Characteristic 

Full 

Sample 

BMCC 

Hostos 

Among those currently employed 
Number of hours worked per week in current job (%) 

1-10 hours 

6.4 

6.2 

7.3 

11-20 hours 

23.4 

23.3 

23.6 

21-30 hours 

22.4 

21.1 

27.0 

31-40 hours 

44.9 

46.8 

38.2 

More than 40 hours 

3.0 

2.7 

3.9 

Average hourly wage at current job ($) 

11.0 

11.2 

10.5 

Highest grade completed (%) 

1 Oth grade or lower 

16.0 

15.9 

16.2 

1 1 th grade 

12.8 

13.2 

11.4 

12 th grade 

65.0 

64.9 

65.5 

Missing 

6.2 

6.0 

6.9 

Diplomas/degrees earned 13 (%) 

High school diploma 

65.0 

65.5 

63.4 

General Educational Development (GED) certificate 

33.1 

32.4 

35.7 

Occupational/technical certificate 

13.3 

13.2 

13.8 

Associate's degree or higher 

2.6 

2.0 

4.6 

None of the above 

2.4 

3.0 

0.3 

Date of high school graduation/GED receipt (%) 

During the past year 

5.2 

5.0 

5.7 

Between 1 and 5 years ago 

29.2 

29.9 

27.0 

Between 5 and 10 years ago 

37.5 

37.4 

38.1 

More than 1 0 years ago 

19.6 

19.0 

21.6 

Missing 

8.5 

8.7 

7.5 

Main reason for enrolling in college 13 (%) 

To complete a certificate program 

2.9 

2.1 

5.9 

To obtain an associate's degree 

50.7 

51.1 

49.2 

To transfer to a 4-year college/university 

43.1 

43.4 

41.8 

To obtain/update job skills 

3.2 

3.0 

4.0 

Other 

1.6 

1.6 

1.9 

First person in family to attend college (%) 

32.9 

31.8 

36.9 

Language other than English spoken regularly in home (%) 

54.6 

49.7 

72.0 

Sample size 

1,502 

1,169 

333 


(continued) 
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Table 1.1 (continued) 


SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 

NOTES: BMCC is an abbreviation for Borough of Manhattan Community College. Hostos is an 
abbreviation for Hostos Community College. 

Characteristics shown in italics are calculated for a proportion of the full sample. 

Missing values are only included in variable distributions for characteristics with more than 5 percent 
of the sample missing. 

Distributions may not add to 100 percent because of rounding. 

“Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who are not Hispanic and chose more than one race are considered multiracial. "Other" 
includes American Indian/Alaskan Native, multiracial, and other. 

b Distributions may not add to 100 percent because categories are not mutually exclusive. 


The sample at both colleges was primarily female. More than half of the sample was 
between the ages of 22 and 26 and the majority was unmarried. Close to half of the sample 
members had at least one child. Nearly all of the sample members were financially independent 
from their parents, as expected based on the eligibility criterion that students live away from 
home. Around 56 percent of sample members were working at the time of random assigmnent. 
Of those who were working, around 70 percent worked an average of 21 or more hours per 
week. Around 65 percent of the sample had completed twelfth grade, and around the same 
proportion had received a high school diploma. Half of the sample members said that the main 
reason for enrolling in college was to obtain an associate’s degree, and 43 percent wanted to 
transfer to a four-year college. Around one-third of sample members said that they were the first 
person in their family to attend college. 

While the samples at both colleges were similar in many aspects, there were some im- 
portant differences. At Hostos, 68 percent of the sample was Hispanic, compared with 38 
percent at BMCC. Likely because of this difference, 72 percent of the sample at Hostos reported 
speaking a language other than English at home, versus 50 percent at BMCC. Sample members 
at Hostos were more likely to be female (79 percent at Hostos versus 66 percent at BMCC) and 
were an older population. Sample members at Hostos were more likely to be married (24 
percent at Hostos versus 17 percent at BMCC) and more likely to have children (59 percent at 
Hostos versus 45 percent at BMCC). Finally, at Hostos, almost half of the sample reported that 
a member of their household was receiving government benefits such as food stamps, cash 
assistance, or welfare, compared with around 35 percent at BMCC. 

Appendix Table A. 1 shows the same demographic characteristics as those reported in 
Table 1.1 for the full sample, program group, and control group. An asterisk in the last column 
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The Performance-Based Scholarship Demonstration 

Table 1.2 

Selected Characteristics of Colleges and Research Samples 


New York Final Report 



Borough of Manhattan 
Community College 

Hostos 

Community College 

Characteristic 

All 

Students 

Research 

Sample 

All 

Students 

Research 

Sample 

Total number of students 

21,858 

1,169 

5,532 

333 

Gender (%) 

Male 

40.2 

33.7 

30.4 

21.0 

Female 

59.8 

66.3 

69.6 

79.0 

Age under 25 years (%) 

67.8 

40.2 

61.0 

32.4 

Race/ethnicity (%) 

Hispanic 

32.9 

37.5 

56.9 

67.7 

White 

14.2 

7.5 

2.0 

1.2 

Black 

33.1 

39.5 

27.9 

29.0 

Asian or Pacific Islander 

11.4 

12.3 

2.8 

0.9 

Other 1 * 

8.5 

3.2 

10.3 

1.2 

Financial aid b 

Received any financial aid (%)° 

78.0 

98.9 

83.0 

96.7 

Any grant aid 

77.0 

98.9 

82.0 

96.7 

Federal Pell Grant 

69.0 

98.0 

78.0 

94.3 

Student loan aid 

7.0 

19.8 

3.0 

6.6 


SOURCES: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System (IPEDS); MDRC calculations using Baseline Information Form 
(BIF) data; MDRC calculations from CUNY Institutional Research Database financial aid data. 

NOTES: Data on all students at both colleges are for undergraduate students from fall 2008. 

Missing values are not included in individual variable distributions. 

Distributions may not add to 100 percent because of rounding, 
includes nonresident aliens, American Indians, and Alaskan Natives. 

b Financial aid data for all students at both colleges are based on first-time, full-time, degree/certificate- 
seeking undergraduates, and represent the 2008-2009 academic year. Financial aid data for the research 
sample at both colleges are based on the first semester in the program only and reflect financial aid 
awarded, not necessarily disbursed. 

c Distributions may not add to 100 percent because categories are not mutually exclusive. 
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of the table indicates that the percentage of program group members with that characteristic is 
significantly different from the percentage of control group members. The small number of 
differences shown between the two research groups could be expected to occur randomly. 14 

Table 1.2 compares the characteristics of the PBS New York sample with characteris- 
tics of all of the undergraduate students at each of these institutions. The comparison shows that 
PBS sample members were more likely to be female and were older than other undergraduate 
students at each college. Sample members were also more likely to be Hispanic and slightly 
more likely to be African-American. Sample members were more likely to be awarded financial 
aid, more likely to be awarded the federal Pell Grant, and more likely to be awarded student 
loans. In sum, the PBS program targeted a particular segment of the student body, and the 
results cannot be generalized to all students who enrolled at these institutions. 


l4 An omnibus test was conducted to assess whether overall systematic differences in baseline characteris- 
tics were observed between the two research groups. The model’s likelihood ratio test yielded a p-value of 
0.47. Convention suggests that this probability is large enough that these potential differences can be ignored in 
the analyses. In addition, characteristics of the PBS and PBS-plus-summer groups were compared. The 
omnibus test yielded a p-value of 0.91 . 
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Chapter 2 

The Implementation of the PBS Program in New York 


At first glance, the program in New York seems relatively simple — a scholarship program 
contingent on meeting academic benchmarks with no other components. As with many pro- 
grams, though, implementing the program proved more complex. How would staff know 
whether students had earned the award? How would payments be triggered and delivered to 
students? To what extent would students take advantage of the scholarship dollars available to 
them? 


Overall, the findings suggest that the PBS program received a “fair test”: (1) the schol- 
arship was delivered as intended over the duration of the study; (2) there was a high rate of 
participation among students in the program group; and (3) students in the control group did not 
have access to similar sources of funds. 

The College Settings 

The majority of sample members attended Borough of Manhattan Community College 
(BMCC). The college itself boasts a claim on four city blocks along the Hudson River, covering 
more than four acres of property in Manhattan’s densely populated downtown core. It is the 
only community college in Manhattan and the largest in the City University of New York 
(CUNY) system, serving over 20,000 students in the fall 2008 semester alone. Attending classes 
in the heart of Manhattan’s vibrant center, BMCC students are surrounded by distinct neighbor- 
hoods ranging from the artistic communities of Tribeca and Soho to the bustling downtown 
financial district. Students come from the New York City area and 155 countries around the 
world. BMCC graduates students from a diverse range of races and ethnicities, with a third of 
graduates being African-American and another third Hispanic. Courses in over 60 subjects lead 
students to a two-year professional technical degree, provide them with continuing education 
opportunities, or prepare them to pursue a degree at a four-year institution. The college also 
serves a large population of developmental education students, with over 85 percent of incom- 
ing first-year students in fall 2008 requiring at least one level of developmental education in 
reading, writing, or mathematics. 1 

Hostos Community College, named to honor the legacy of Puerto Rican educator, writer, 
and patriot Eugenio Maria de Hostos, occupies several buildings in the Bronx, all clustered at 

'See www.bmcc.cuny.edu and www.bmcc.cuny.edu/iresearch/upload/Fall08FactSheet.pdf (“BMCC Fact- 
sheet Fall 2008”). 
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149th Street and the Grand Concourse, in contrast to the sprawling campus of BMCC. The 
college was established in the Bronx to meet the educational needs of people from the area and 
similar communities who have historically been excluded from higher education. Carrying 
forward its namesake’s life commitment to expand educational opportunities to underserved 
populations, Hostos educates students, mainly Bronx residents, from diverse ethnic, racial, 
cultural, and linguistic backgrounds. It serves a greater proportion of its local community than 
does BMCC. It is also much smaller than BMCC, enrolling around 5,500 students in the fall 2008 
semester. The majority of the student population is Hispanic (around 60 percent), with African- 
American students making up another 30 percent. As at BMCC, a large proportion (89 percent) of 
incoming first-year students in fall 2008 required at least one level of developmental education. 2 

Scholarship Policies and Procedures 

As noted in the previous section, scholarships were paid in three installments during the fall and 
spring semesters and two installments in the summer semester. At BMCC, the program was 
housed in the Office of Financial Aid, while at Hostos, it was housed in the Office of Student 
Development and Enrollment Management. However, the payment process differed only 
slightly at the two colleges. MDRC visited both BMCC and Hostos staff for a quality assurance 
check of their scholarship disbursement process. The visits were infomiative and revealed the 
college staff members’ effectiveness, dedication, and strong attention to detail. 3 

At BMCC, at each payment interval, financial aid staff created reports on the students in 
the program group, detailing transcript information for that semester (including grades and credits 
in each course). The site coordinators then manually went through the list of students to determine 
whether each student met the criteria to earn a scholarship. Once this task was complete, the 
payment amount was entered into the system, triggering the check disbursement or direct deposit 
from the Central Office system at BMCC. Since the check could possibly represent other funds in 
addition to the scholarship (such as a refund), site coordinators also sent a letter to students that 
outlined the amount of their expected disbursement and the date that students should expect it to 
arrive. This step was intended to help keep the scholarship salient to students. 

At Hostos, at each payment interval, the site coordinator examined academic transcripts 
for program group students to determine each student’s eligibility for a payment. Then the 
amount and date of disbursement were entered into the Student Financial Aid system and 


2 See Hostos Web site (www.hostos.cuny.edu) and “Hostos Student Profiles” for fall 2008 term 
(www.hostos.cuny.edu/oaa/oir/PublicDocuments/StudentProfile.pdf). 

3 MDRC additionally cross-referenced scholarship payment data and transcript data to determine whether 
students were paid correctly. While a handful of errors did occur, the error rate was approximately 1 percent, 
and all errors resulting from underpayments were corrected. 
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cleared for distribution. As at BMCC, the site coordinator sent a letter to students letting them 
know the date and amount of the payment they could expect. 

Staff at both colleges communicated with program group students at each payment 
point during each semester that they were eligible for a scholarship. In cases where students 
were awarded a scholarship payment, the letter offered congratulations; when students were not 
awarded scholarship funds, the letter reminded them that they were still eligible for future 
payments. Coordinators at both colleges were available to answer students’ questions by e-mail 
or phone. 

At both colleges, the programs were not intended to have a counseling component; thus, 
it was expected that the scholarship payment staff would restrict their interactions with students 
to answering questions related to financing students’ educations. Given that the scholarship 
program at BMCC was completely managed by the financial aid office, students were not likely 
to receive any additional guidance or counseling from program staff. When students visited the 
financial aid office, the staff could generally only answer questions around students’ financial 
aid packages, such as when their money would arrive and how much they could expect to 
receive. The actual office is arranged similarly to a bank, where students line up in rows and 
wait for attention from a “teller.” In contrast, when students visited the student services division 
at Hostos, they could access any number of the services offered at the college. The division 
handles inquiries regarding career services, social services, and academic advising, and also 
provides access to the financial aid office and the registrar. The office has a seating area with 
magazines and a front desk assistant who can direct students to any one of the services provid- 
ed. There is suggestive evidence that students at Hostos received additional ad hoc advising, 
since the coordinator was part of the student services division and located in a central area of the 
campus. 


BMCC also had three times as many students to manage in the program (587 program 
group members at BMCC, versus 167 at Hostos). Program staff at Hostos may have found it 
easier to manage the smaller number of students, may have interacted more with students 
because the program was housed in student services, and may have had time to provide addi- 
tional support to students owing to the smaller number of students in the program overall. 
Unfortunately, although interviews with program staff are usually a main source of data for 
such details on implementation, MDRC was not able to conduct such interviews for this 
program. Thus, more infomiation on how the program was implemented at each college, 
whether this implementation differed, and to what extent Hostos students may have received 
additional support is unknown. 
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Participation in the Performance-Based Scholarship Program 

Figure 2.1 details the scholarship payment rates for the duration of the program. 4 In the first 
semester, 98.8 percent of students received the first payment and 96.9 percent of students 
received the second payment. Given that students were already registered at the time that they 
entered the study, it was expected that nearly all program group students would receive the 
initial and midterm payments in the first semester of the program. Around 72 percent of 
program group students went on to earn the final payment in the first semester of the program, 
earning the full $1,300 award. Among those program group students who earned any award, the 
average scholarship amount for the semester was $1,114. 

In the second semester, 83.2 percent of program group students enrolled in six or more 
credits, earning them the initial payment of $200. A similar proportion earned the midterm 
payment, and then a lower proportion earned the final payment contingent on grades (57.8 
percent). Overall, in the second semester, among those program group students who earned any 
award, the average scholarship amount for the semester was $1,084. 

The PBS-plus-summer group was eligible for an additional $1,300 summer scholarship. 
In this group, 31.5 percent of students earned an initial payment of $200, and 25.1 percent 
earned the final payment (13.8 percent earned the part-time award and 1 1.4 percent earned the 
full-time award). Notably, the majority of PBS-plus-summer group members did not earn the 
summer scholarship because they did not register for summer courses, leaving valuable scholar- 
ship dollars on the table. Summer courses can often require two or three times as many in-class 
hours per week as those offered in fall or spring, and this may have posed an added constraint 
for students. It is likely that sample members did not attend for reasons outside of their direct 
control — students who are parents, for example, may not have child child care in the summer, 
or working students may have larger commitments over the summer. Limited course availabil- 
ity in the summer may also be a factor. 

The Contrast Between PBS and Regular Services 

To increase the possibility that the program would act as an incentive, not only must students 
participate in the program, but the program needs to be distinct from other programs available to 
them. In the case of this scholarship-only test in New York, this means that students in the 

4 Less than 1 percent of the sample did not receive the initial and/or the midterm payment on the scheduled 
date, but recouped the appropriate payments at the end of the semester. In addition, less than 1 percent of the 
sample did not receive the full amount “earned" because the sample members owed an outstanding balance to 
the college. In this figure, these sample members are counted as having received each of these payments. 
Approximately 1 percent of program group students received a payment that they were not eligible for and 
should not have received. 
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The Performance-Based Scholarship Demonstration 
Figure 2.1 

Scholarship Receipt Among Program Group Members 
New York Final Report 
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SOURCE: MDRC calculations from Borough of Manhattan Community College and Flostos Community College 
scholarship payment data. 

NOTES: Less than 1 percent of the sample did not receive the initial and/or the midterm payment on the 
scheduled date, but recouped the appropriate payments at the end of the semester. In addition, less than 1 percent 
of the sample did not receive the full amount "earned" because the sample members owed an outstanding balance 
to the college. In this figure, these sample members are counted as having received each of these payments. 

Approximately 1 percent of program group students received a payment that they were not eligible for and 
should not have received. 


21 


program group must receive more financial aid than those in the control group, on average, and 
the increase in their financial aid must be sizable enough to influence behavior. Financial aid 
data collected from CUNY’s central offices provided valuable information on the amount and 
source of financial aid awarded to the students in the sample. 5 

As mentioned, the performance-based scholarship was available to members of the pro- 
gram group only, while both program and control group members were potentially eligible for 
federal Pell Grants, New York State’s Tuition Assistance Program, and other fonns of financial 
aid. The performance-based scholarship was intended to increase the overall amount of money 
available to students in the program group. 

Table 2.1 presents the impact of the perfomiance-based scholarship program on financial 
aid packages. (See Box 2.1 for an explanation of how to interpret an impact table.) The first panel 
shows the results for the first program semester. Almost all program group students were awarded 
the perfonnance-based scholarship. 6 Almost all students in both groups were awarded the Pell 
Grant, an eligibility condition for inclusion in the demonstration. Almost half of students were 
awarded New York State’s Tuition Assistance Program (TAP) grant, and a similar proportion 
were awarded some other grant. Fewer than one-fifth of students were awarded a subsidized loan, 
and even fewer were awarded an unsubsidized loan. In total, students in the program were 
awarded an average of $4,327 in financial aid in the first program semester, compared with 
$3,136 for students in the control group (a difference of $1,191, significant at the 1 percent level). 
This difference is almost entirely due to the perfonnance-based scholarship. 

The second panel shows similar information for the second program semester of the 
scholarship. Again, the vast majority of the program group continues to have the perfonnance- 
based scholarship, and the majority of the sample continues to have the Pell Grant in their 
financial aid package. The overall aid package is approximately $1,018 higher, on average, for 
students in the program group, again owing wholly to the perfonnance-based scholarship. 

Student Perceptions and Experiences 

As noted in the section on data sources, individual interviews were conducted with about 20 
students to leam about their experience in the study. In April of 2010, MDRC randomly 
selected 50 program group students and 50 control group students from each college in the 


5 Data collected represent amounts awarded to students, not necessarily disbursed to students. 

6 Outcomes shown for the performance-based scholarship are calculated based on financial aid data pro- 
vided by CUNY, not scholarship payment data provided by the colleges. For BMCC, the awarded amounts 
shown for the performance-based scholarship were manually updated by the site coordinators to reflect actual 
amounts paid. At Hostos, no such updates were made. 
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The Performance-Based Scholarship Demonstration 
Table 2.1 

Impacts on Financial Assistance During Program Semesters 


New York Final Report 


Outcome 

Program 

Group 

Control Difference Standard 

Group (Impact) Error 

First program semester 





Awarded any financial assistance (%) 

98.9 

97.9 

1.1 * 

0.6 

Performance-based scholarship' 1 

98.8 

0.0 


0.4 

Federal Pell Grant 

97.1 

97.3 

-0.3 

0.8 

TAP grant 

47.9 

45.7 

2.2 

2.6 

Other grants’ 5 

45.6 

47.2 

-1.6 

2.5 

Subsidized loans 

16.8 

16.8 

0.0 

1.9 

Unsubsidized loans 

8.4 

8.8 

-0.5 

1.4 

Average financial assistance awarded ($) 

4,327 

3,136 

1,191 *** 

90 

Performance-based scholarship 3 

1,120 

0 

1,120 *** 

12 

Federal Pell Grant 

1,974 

1,935 

39 

35 

TAP grant 

605 

567 

37 

34 

Other grants’ 5 

109 

121 

-12 

9 

Subsidized loans 

326 

316 

10 

39 

Unsubsidized loans 

193 

196 

-4 

36 

As grants/scholarships (%) 

92.3 

90.6 



As loans (%) 

7.7 

9.4 



Second program semester 





Awarded any financial assistance (%) 

85.8 

81.9 

2 g ** 

1.9 

Performance-based scholarship' 1 

83.3 

0.0 

83.3 *** 

1.4 

Federal Pell Grant 

79.9 

78.7 

1.2 

2.1 

TAP grant 

38.7 

35.6 

3.2 

2.5 

Other grants’ 5 

37.9 

40.6 

-2.7 

2.5 

Subsidized loans 

17.1 

16.7 

0.4 

1.9 

Unsubsidized loans 

8.9 

9.4 

-0.5 

1.5 

Average financial assistance awarded ($) 

3,722 

2,704 

1,018 *** 

116 

Performance-based scholarship 3 

930 

0 

930 *** 

18 

Federal Pell Grant 

1,661 

1,600 

61 

52 

TAP grant 

508 

472 

36 

35 

Other grants’ 5 

100 

111 

-12 

8 

Subsidized loans 

323 

303 

20 

38 

Unsubsidized loans 

200 

218 

-17 

38 

As grants/scholarships (%) 

90.7 

88.1 



As loans (%) 

9.3 

11.9 



Sample size ( total = 1,502 ) 

754 

748 
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(continued) 







Table 2.1 (continued) 


SOURCE: MDRC calculations from CUNY Institutional Research Database financial aid data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between the research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and college. 

Outcomes shown in italics indicate nonexperimental data. Significance tests are not calculated 
for nonexperimental data. 

This table shows results for the first and second program semesters for the performance-based 
scholarship available to all program group students and contains data from fall and spring semesters 
only. Summer semester data are not shown. Federal Work-Study awards were only reported for 
summer terms at both colleges, and are not presented in the table. 

“Outcomes shown for the performance-based scholarship are calculated based on financial aid 
data provided by CUNY, and not scholarship payment data provided by the colleges. For Borough 
of Manhattan Community College, the awarded amounts shown for the performance-based 
scholarship were manually updated by the site coordinators to reflect actual amounts paid. At 
Hostos Community College, no such updates were made. 

b This includes all grants and scholarships excluding the federal Pell Grant, TAP grant, 
and performance-based scholarship awards. 


spring 2009 cohort. These students were contacted and invited to participate in an interview. A 
similar process was completed in September of 2010 for students from the fall 2009 cohort. The 
interview pool contained a mix of those in the program and control groups. However, the 
students who came forward for these interviews represent a small portion of the overall sample; 
in addition, these students tend to be among the most responsive and motivated. Thus, caution 
should be exercised when interpreting the findings and results from these interviews. 

During the interviews, MDRC learned about students’ employment, family life, orienta- 
tion to college, finances, and motivation to achieve the scholarship’s academic benchmarks. Of 
the students interviewed, there was a mix of students who had attended college or other 
postsecondary institutions prior to attending at BMCC or Hostos and students who had not 
attended college before these two colleges. Most of the students interviewed combined school 
and employment. Students were also at different points in their education, including having 
graduated or transferred to another institution by the time the interview was conducted. 

Orientation to College 

Interviews delved into students’ experiences in pursuing their college education, asking 
specifically about obstacles and challenges they have faced. Most students mentioned time 
management and finding time for school. One program group student described the difficultites 
of finding a balance between school and work: 
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Box 2.1 


How to Read the Impact Tables in This Report 

Most tables in this report use the format illustrated in the abbreviated table below, which 
displays some hypothetical transcript data for the program and control groups. The table shows 
that program group students attempted an average of 37.7 credits and control group students 
attempted an average of 36.0 credits. 

The “Difference” column in the table shows the observed difference in outcome between the 
two research groups — that is, the estimated average impact of the opportunity to participate in 
the program. For example, the estimated average impact on credits attempted can be calculated 
by subtracting 36.0 from 37.7, yielding an impact estimate of 1.7 credits attempted. This 
difference represents the estimated average impact rather than the true average impact (which 
is impossible to determine) because, although study participants are randomly assigned to the 
program and control groups, the impact estimate would have been different if a different 
sample of students had been included in the study or if the same group of students had been 
randomized in a different way. 

Differences marked with one asterisk or more are considered statistically significant, meaning 
that there is a high probability that the opportunity to participate in the program had an impact 
on that outcome measure. Differences that have no asterisk indicate that the opportunity to 
participate in the program did not have a discernible effect on that outcome. The number of 
asterisks indicates the probability that an impact at least as large as the one observed in the 
study would have occurred even if the true average impact had been zero. One asterisk corre- 
sponds to a 10 percent probability; two asterisks, a 5 percent probability; and three asterisks, a 
1 percent probability. The more asterisks, the more likely that the opportunity to participate in 
the program had a true average impact on the outcome. The impact in the table excerpt below 
has one asterisk, indicating that the impact is statistically significant at the 10 percent level — 
meaning that there is a 10 percent chance of observing an estimated average impact this large 
(or larger) if the opportunity to participate in the program actually had no average effect on 
credits attempted. In other words, there is a 90 percent level of confidence that the opportunity 
to participate in the program had a positive impact on the average number of credits attempted. 

Also shown in the table is the standard error of the impact estimate. The standard error is a 
measure of uncertainty or variability around the impact estimate. Some useful rules of thumb 
are that there is about a 90 percent chance that the true average impact is within plus or minus 
1 .65 standard errors of the estimated average impact, roughly a 95 percent chance that the true 
average impact is within plus or minus 1.96 standard errors of the estimated average impact, 
and about a 99 percent chance that the true average impact is within plus or minus 2.58 stand- 
ard errors of the estimated average impact. For example, in the data below, there is roughly a 
90 percent chance that the average impact on credits attempted lies between 0.2 and 3.2, 
calculated as 1.7 ± (1.65 x 0.9). 



Program 

Control 


Standard 

Outcome 

Group 

Group 

Difference 

Error 

Number of credits attempted 

37.7 

36.0 

1.7* 

0.9 
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For a while I was working full time....l don’t know how people do it. f don’t 
think I could ever do it again. And a lot of students here at the school do it and it 
is so hard. It is so hard. That was probably one of the challenges I’ve had as far 
as trying to juggle work and school. 

Other qualitative research suggests that achieving balance between work, school, and 
family demands is a challenge that many community college students face . 7 The question for 
this evaluation is whether a perfomiance-based scholarship ameliorates some of the financial 
struggle and helps students perfonn better in school. 

Other challenges cited by students included professor-student relationships, handling 
college without a support system, being a single parent, having to take remedial coursework, 
and coming from another country. It was clear that navigating college was not a simple task for 
the sample members interviewed, though some had found methods to make things work. 

Financial Aid and the Cost of Attending College 

Most students interviewed reported moderate or little concern about paying for college 
specifically, but acknowledged that grants covered most of the cost of their education. One 
control group student noted in particular that attending community college helped her manage 
the cost of attending school. The student went on to explain that financial aid helped her with 
these costs: 


In seeking financial aid this time in school it. ..somewhat empowered me be- 
cause I was able to go at the pace I needed to go financially and I was also able 
to cover the cost on my own as opposed to the first time around in school having 
my parents paying a lot of money and not complete. This way as an adult student 
it was just like declaring and taking ownership and taking claim of my education. 

Students generally viewed financial aid as a significant facilitator of college access, with- 
out which they could not afford to attend college. Many students interviewed reported receiving 
financial assistance from family or friends. Some students mentioned prior college experiences 
where they took out a loan to pay for college, but generally students aimed to avoid debt. 

Significance of Performance-Based Scholarship and Motivation to 

Perform 

Across both program and control students, there was a view that the perfonnance-based 
scholarship was a good idea and that the program was beneficial to students. Control group 


7 Matus-Grossman et al. (2002); Nelson and Purnell (2003). 
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students generally acknowledged that the scholarship would have made it easier to cover 
additional expenses, had they received it. However, there were mixed responses from program 
group students on how the scholarship affected their performance. In general, there was a 
feeling that students did not have to do much out of the norm to meet the perfomiance bench- 
marks, and that they only thought about the scholarship on occasion throughout the semester. 
The impression the students gave was that the scholarship was not meaningful to them in terms 
of changing behavior, though one student said that the scholarship motivated her to get higher 
grades and spend more time studying. 

Students were also asked how the scholarship affected their course-taking behavior; one 
student responded: 

I don’t think it did. . ..It’s not like 1 sat there and thought, “Oh, now 1 have schol- 
arships so I’m going to take, you know, this class or that class.” I kinda look at it 
as I’m just going to take classes that I need to take and that I want to take as far 
as completing my degree. 

That said, most program group students reported that the scholarship helped to reduce 
financial pressure and worry. 

The findings from the one-on-one interviews with students were also consistent be- 
tween the two colleges, with one exception. The scholarship program at Hostos was called 
“ScholarBound,” whereas the program at BMCC was called “Performance-Based Scholarship.” 
At Hostos, the name “ScholarBound” was received positively by the program group students 
who were interviewed; however, students were not prompted to comment on the scholarship 
name at BMCC. 

Summary of Individual Interviews 

While the individual interviews confinn the challenges facing adult learners — includ- 
ing working full time, handling college without a support system, and managing financial aid — 
the overall impression was that the scholarship did not have a large influence on students’ 
thinking and behavior. Most students did not feel that they had to change their behavior in order 
to meet the performance benchmarks, but thought about the scholarship on occasion throughout 
the semester. 
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Chapter 3 

Academic Impacts 


A key goal of the Performance-Based Scholarship (PBS) Demonstration is to test whether 
performance-based scholarships help students perform better in college. The academic findings 
presented in this section follow students for four main semesters after random assignment (main 
semesters are considered fall and spring semesters), and include any winter and summer 
sessions in between. 

The main findings of this follow-up report are: 

• The scholarship-only program encouraged more full-time enrollment 
during the program semesters. Full-time enrollment increased 4.1 percent- 
age points in the first semester and 6.0 percentage points in the second se- 
mester of the scholarship program. 

• The scholarship-only program did not increase the average number of 
semesters registered or credits earned for the full sample over the two- 
year follow-up period. While program group students attempted more cred- 
its on average, this impact was fairly small and was not accompanied by an 
impact on the number of semesters registered. This impact also did not trans- 
late into an impact on cumulative credits earned. 

• There is evidence that offering the scholarship over the summer term 
increased registration and credit accumulation in that semester. Howev- 
er, these effects did not affect cumulative outcomes. 

• There is exploratory evidence that the program had impacts on students 
at one college but not on students at the other college, and that, in the 
former case, the program had positive impacts on important markers of 
academic success. In light of a desire to give feedback to the colleges, the 
impacts of the scholarship-only program were estimated separately for each 
college. Such evidence is exploratory in nature as this subgroup analysis was 
not designated ahead of time. 1 There are significant impacts at one of the col- 


'Schochet (2008) suggests that the purpose of exploratory analysis is to examine relationships within the 
data to identify outcomes or subgroups for which impacts may exist. Subgroups specified in advance can be 
part of the confirmatory analyses, but those analyzed post hoc (such as the college subgroup discussed here) are 
not part of the confirmatory analysis and should be treated as exploratory. 
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leges on registration, full-time enrollment, and credit accumulation, and these 
effects persist after the scholarship ends. 

• The total cost per student at the two colleges was roughly $3,000. This to- 
tal cost comprised around $2,100 in scholarship costs and $900 in support 
and administration. 

This chapter begins with detailed findings around the study’s primary analysis, and is 
followed by a look at some secondary analyses. 

Primary Analyses 

Table 3.1 outlines the key academic outcomes among sample members over the follow-up 
period. (See Box 3.1 for an explanation of how the semesters are combined.) On average, 
program and control group members registered for the same number of semesters (around 
three). 2 Program group students attempted 1.7 more credits, on average, than control group 
students, and this difference is statistically significant at the 10 percent level. However, this 
impact did not translate into an impact on credits earned. While program group students earned 
26.7 credits on average and control group students earned 25.6 credits on average, this differ- 
ence is not statistically significant. Similarly, there are no statistically significant findings on 
cumulative grade point average over the same time period. 

An earlier report on the partial sample revealed impacts on full-time enrollment in the 
first and second program semesters and small impacts on credits attempted and earned in the 
first program semester; these impacts are maintained when looking at the full sample. Appendix 
Table B.l details the academic impacts on the full sample by semester. There is a 4.1 percent- 
age point impact on full-time enrollment in the first semester and a 6.0 percentage point impact 
on full-time enrollment in the second semester. Additionally, program group students attempted 
and earned about a half credit more than control group students in the first semester. However, 
these early impacts in individual semesters do not continue in the absence of the scholarship, 
nor do they lead to an impact on credits earned over the cumulative period. This suggests that 
while the program was effective when students were eligible for scholarships, the enrollment 
and credits earned effects dissipated after the program ended. 

Figure 3.1 details the credit accumulation during the first four semesters after random 
assignment. At the end of the first semester, there is an impact of about half a credit, significant 


2 Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits and 
grades are not included in the outcomes shown in this report. Enrollment figures and credits attempted are thus 
somewhat lower than they would be otherwise. 
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The Performance-Based Scholarship Demonstration 
Table 3.1 

Cumulative Academic Outcomes Among Sample Members: 
First Through Fourth Semesters 


New York Final Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Number of semesters completed 

3.1 

3.1 

0.1 

0.1 

Average number of credits attempted 

37.7 

36.0 

1.7 * 

0.9 

College-level credits 

27.6 

26.3 

1.3 

0.8 

Developmental credits 

10.1 

9.8 

0.3 

0.4 

Average number of credits earned 

26.7 

25.6 

1.1 

0.9 

College-level credits 

22.0 

20.8 

1.2 

0.8 

Developmental credits 

4.7 

4.8 

-0.1 

0.3 

Cumulative GPA (%) 

3.0 to 4.0 

32.2 

35.6 

-3.3 

2.4 

2.0 to 2.9 

39.0 

36.4 

2.6 

2.5 

Less than 2.0 

23.1 

21.8 

1.3 

2.2 

No GPA a 

5.7 

6.3 

-0.6 

1.2 

Sample size (total = 1,502 ) 

754 

748 




SOURCE: MDRC calculations from Borough of Manhattan Community College and Hostos 
Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and college. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their 
credits and grades are not included in the outcomes shown in this table. 

A few college-level courses have excess "compensatory credits" associated with them. These 
credits are used to calculate total credit hours, but do not count toward a degree. In this table, 
compensatory credits are included in the developmental credits category and make up less than 1 
percent of those credits. 

a The "No GPA" category includes students who are not enrolled and students who took only 
non-letter-graded courses, which are not included in GPA calculations. 

at the 5 percent level. At the end of the second semester, there is an impact of about a credit, 
significant at the 10 percent level. However, by te end of the third and fourth semesters, the 
control group level of credits earned becomes similar to that of the program group students. 
Appendix Table B.2 provides more details regarding the impacts and standard errors for these 
outcomes. 
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Box 3.1 


Timing of Academic Semesters 

At the Borough of Manhattan and Hostos community colleges, the academic year comprises two 
main semesters (fall and spring), one winter session, and two summer sessions. The fall semester 
stretches from late August to mid-December, followed by a one -month winter session in January. 
The spring semester goes from late January through May, followed by two summer sessions of 
around one month each in June and July. 

Throughout this report, the follow-up period is four main semesters, and cumulative outcomes 
include any winter and summer sessions in between. As a result of institutional data reporting 
methods, any credits attempted or earned in the winter session are combined with those attempted or 
earned during the fall semester; these credits are not reported separately. However, since the summer 
session was of particular interest given the summer component of the scholarship, it is not combined 
with any other semesters. Thus, Table 3.2 shows the summer semester alone, and combines the 
results from the two summer sessions. In Appendix Table B.l, Appendix Table B.4, and Appendix 
Table B.5, where results are shown by term, the summer semester is excluded from the first four 
panels (since it is presented separately in Table 3.2). 

As a result of the timing of intake, which was conducted over multiple semesters, the tables in this 
report show different semesters for each cohort. The following table summarizes the cohorts and 
semesters that are covered in this report. (Entries correspond to labels in tables.) 






Calendar Semester 




Cohort 

Fall 

2008 a 

Spring 

2009 

Slimmer 

2009 

Fall 

2009 a 

Spring 

2010 

Summer 

2010 

Fall 

2010 a 

Spring 

2011 

Fall 

2008 

1st 

semester 

2nd 

semester 

1st 

summer 

semester 

3rd 

semester 

4th 

semester 

— 

— 

— 

Spring 

2009 


1st 

semester 

1st 

summer 

semester 

2nd 

semester 

3rd 

semester 

2nd 

summer 

semester 

4th 

semester 


Fall 

2009 

— 

— 

— 

1st 

semester 

2nd 

semester 

1st 

summer 

semester 

3rd 

semester 

4th 

semester 

Tall semesters include the following winter session. 
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The Performance-Based Scholarship Demonstration 
Figure 3.1 

Total Credit Accumulation During the First Four Semesters 
New York Final Report 
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SOURCE: MDRC calculations from Borough of Manhattan Community College and Hostos Community 
College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and college. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits are 
not included in the outcomes shown in this figure. 
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Additionally, Box 3.2 provides details on the cost of the performance-based scholarship 
program at the two colleges. 

Analysis of the Summer Term 

As mentioned, half of the program group members were eligible to receive a summer 
scholarship. Table 3.2 reports on the analysis of outcomes for those in the PBS-plus-summer, 
PBS, and control groups. The first three columns of this table show the average outcome levels for 
the three research groups. These columns are followed by column s that present comparisons of 
outcomes for these three groups. The first panel of the table details outcomes for the first summer 


Box 3.2 

Cost of Performance-Based Scholarships 

The cost of the Performance-Based Scholarship program at two New York colleges was about 
$3,000 per program group member, 70 percent ($2,100) of which pays for scholarships and 30 
percent ($900) of which pays for program support and administration. The scholarship costs repre- 
sent the amount of money the program paid directly to students. Support and administration includes 
all other costs such as salaries, benefits, overhead associated with program staff, supplies, materials, 
travel, and professional development. 

The cost of support and administration should be viewed as a relatively fixed cost that is likely to 
decrease as the program serves more students. This claim is supported by the observed cost at the 
respective schools. Specifically, the cost of support and administration per student at Hostos Com- 
munity College, where the sample size was small (167 program group members), was much higher 
than the cost of support and administration per student at the Borough of Manhattan Community 
College (BMCC), where the sample was larger (587). Such economies of scale would likely contin- 
ue to be realized if the program were to be expanded. That is, the support and administration cost per 
student is likely to decrease as the number of students in the program increases. 


Cost per Student 

New York Program 

BMCC 

Hostos 

Scholarship ($) 

2,100 

2,020 

2,380 

Support and administration ($) 

900 

750 

1,410 

Total ($) 

3,000 

2,770 

3,790 

Number of students in 
program group 

754 

587 

167 
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The Performance-Based Scholarship Demonstration 

Table 3.2 

Academic Outcomes Among PBS, PBS Plus Summer, and Control Groups 


New York Final Report 



Average Outcome Levels 

PBS Plus Summer 
vs. Control 

PBS vs. Control 

PBS Plus Summer 
vs. PBS 

Outcome 

PBS Plus 
Summer 

PBS 

Control 

Group 

Difference 

Standard 

Error 

Difference 

Standard 

Error 

Difference 

Standard 

Error 

First summer semester 

Enrolled as of semester end (%) 

26.2 

19.4 

20.2 

6.0 ** 

2.6 

- 0.8 

2.6 

6.8 ** 

3.0 

Average number of credits attempted 

1.5 

1.1 

1.1 

0 4 ** 

0.2 

0.0 

0.2 

0 4 ** 

0.2 

College-level credits 

1.2 

0.8 

0.9 

0.3 ** 

0.1 

- 0.1 

0.1 

0 4 ** 

0.2 

Developmental credits 

0.2 

0.2 

0.2 

0.0 

0.1 

0.0 

0.1 

0.0 

0.1 

Average number of credits earned 

1.2 

0.9 

1.0 

0.3 * 

0.1 

- 0.1 

0.1 

0.3 * 

0.2 

College-level credits 

1.1 

0.8 

0.8 

0.3 ** 

0.1 

0.0 

0.1 

0.3 ** 

0.2 

Developmental credits 

0.1 

0.1 

0.2 

0.0 

0.1 

0.0 

0.1 

0.0 

0.1 

First through fourth semesters 

Number of semesters completed 

3.1 

3.1 

3.1 

0.1 

0.1 

0.1 

0.1 

0.0 

0.1 

Average number of credits attempted 

37.5 

37.9 

36.0 

1.5 

1.1 

1.9 

1.1 

-0.4 

1.3 

College-level credits 

27.5 

27.7 

26.3 

1.2 

1.0 

1.5 

1.0 

-0.3 

1.2 

Developmental credits 

10.0 

10.2 

9.8 

0.3 

0.5 

0.4 

0.5 

- 0.2 

0.6 

Average number of credits earned 

26.6 

26.8 

25.6 

1.0 

1.1 

1.2 

1.1 

- 0.2 

1.3 

College-level credits 

22.0 

22.0 

20.8 

1.2 

1.0 

1.2 

1.0 

0.0 

1.1 

Developmental credits 

4.7 

4.8 

4.8 

- 0.2 

0.3 

0.0 

0.3 

- 0.2 

0.4 

Cumulative GPA (%) 

3.0 to 4.0 

33.1 

31.4 

35.6 

-2.5 

3.0 

-4.2 

3.0 

1.7 

3.4 

2.0 to 2.9 

37.6 

40.4 

36.4 

1.2 

3.1 

4.1 

3.1 

-2.9 

3.5 

Less than 2.0 

23.8 

22.3 

21.8 

2.0 

2.6 

0.5 

2.6 

1.5 

3.0 

No GPA a 

5.5 

5.9 

6.3 

-0.7 

1.5 

-0.4 

1.5 

-0.3 

1.7 

Sample size ( total = 1,502 ) 

378 

376 

748 








(continued) 



Table 3.2 (continued) 


SOURCE: MDRC calculations from Borough of Manhattan Community College and Hostos Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; 
* = 10 percent. 

Estimates are adjusted by research cohort and college. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits and grades are not included in the outcomes 
shown in this table. 

A few college-level courses have excess "compensatory credits" associated with them. These credits are used to calculate total credit hours, but do not 
count toward a degree. In this table, compensatory credits are included in the developmental credits category and make up less than 1 percent of those 
credits. 

a The "No GPA" category includes students who are not enrolled and students who took only non-letter-graded courses, which are not included in GPA 
calculations. 
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term of the sample; the second panel details cumulative outcomes, similar to that shown in 
Table 3.1, for the sample. 3 

Comparing outcomes for the PBS-plus-summer group and the control group addresses 
the question: What were the effects of a combined regular-year and summer scholarship 
program on summer semester attendance? (See first panel of Table 3.2.) About 26 percent of 
the PBS-plus-summer group registered for summer courses compared with 20.2 percent of the 
control group, a statistically significant impact at the 5 percent level. This 6 percentage point 
difference represents about a 30 percent increase in summer enrollment. There are also small 
but significant impacts on credits attempted and earned overall in the summer. 

Comparing the outcomes for the PBS group and the control group helps to answer the 
question: What were the effects of a regular-year scholarship, by itself on summer attendance? 
There are no statistically significant differences between outcomes for these groups during this 
summer term. 

Finally, comparing the PBS-plus-summer group and PBS group outcomes addresses the 
question: What were the added effects of the summer scholarship on summer attendance? 
Similar to the first comparison, this one yields statistically significant findings on registering in 
the summer, credits attempted, and credits earned. There is a 7 percentage point impact on 
summer enrollment, representing an increase of about 35 percent. 

The second panel of Table 3.2 answers these same three questions, but measures the ef- 
fect on the two years of follow-up. Outcomes shown are identical to those shown previously in 
Table 3.1, but the program group here is separated into the PBS and PBS-plus-summer groups. 
In examining the cumulative academic outcomes over the follow-up period, there is no evidence 
that providing a summer scholarship during one summer increased the cumulative number of 
semesters registered, credits attempted, or credits earned. While the in-program impacts on 
summer attendance are positive and encouraging, they do not translate into meaningful impacts 
on longer-term outcomes. Additionally, there were no impacts on registration in the fall semes- 
ter following the summer semester where students in the PBS-plus-summer group were eligible 
for the summer scholarship (not shown in tables). This suggests that while there were strong and 
relatively large impacts in the summer semester that the PBS-plus-summer group was offered a 
scholarship, these do not persist in the absence of the scholarship offer. 


3 The characteristics of the PBS and PBS-plus-summer groups were also compared. The omnibus test 
yielded a p-value of 0.91. 
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Secondary Analyses 


Subgroup Analyses 

Although there are no detectable effects on the key academic outcomes of students 
pooled across the two institutions, it is possible that the program was helpful for certain types of 
students. Impacts on educational outcomes for subgroups of students, defined using characteris- 
tics that are measured at or before the onset of the study, were examined for differences in the 
effect of the performance-based scholarship program. 

Impact Variation by Parental Status 

The original test of perfonnance-based scholarships in Louisiana saw large, positive 
impacts for a population of low-income parents. Though the population in New York is differ- 
ent from that in Louisiana, analyses were conducted to determine whether the perfonnance- 
based scholarship program in New York had a differential impact for parents versus nonparents. 
Findings show a statistically significant impact on total credits earned for parents (2.7 credits 
earned on average, over the same follow-up period as shown in Table 3.1). There is a differen- 
tial impact on credits earned for parents compared with nonparents, significant at the 10 percent 
level, meaning that the impact on credits earned for parents is statistically different from the 
impact for nonparents. Considering the period of analysis (close to two academic years), the 
impact detected may seem modest in size (2.7 credits), but it does equate to about a full course 
over this period of time. The detailed table is shown in Appendix Table B.3. The structure of the 
table is similar to that of other tables presented in the report, except that the last column contains 
daggers showing the differences between the impacts for parents and nonparents. 

While the evidence above suggests that the program was more effective for parents than 
nonparents, there were also observed differences in program impacts for the two colleges 
(discussed later in this section). When the parent subgroup is examined at BMCC alone, there 
are no statistically significant impacts for parents or nonparents (not shown in tables). At 
Hostos, there are impacts for both parents and nonparents, and the difference between these two 
impacts is not statistically significant (not shown in tables). Thus, it is likely that since there is a 
higher proportion of parents at Hostos, the differential impacts based on parental status are 
confounded with the differential impacts by college. 

Impact Variation by Other Characteristics at Baseline 

Because some community college outcomes have been found to differ by gender, an 
analysis was conducted to determine whether perfonnance-based scholarships in New York had 
a differential impact by gender. This subgroup analysis found no meaningful significant 
differences in impacts between men and women. Similarly, an analysis was conducted to 
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determine whether the program had differential impacts by ethnicity, and found no meaningful 
significant differences in impacts between Hispanic students and non-Hispanic students. 
Finally, an analysis was conducted to determine whether the program had differential impacts 
by cohort (recall that there were three cohorts of study in New York), and this analysis also 
found no significant differences in impacts between the first, second, and third cohort of study. 

Impact Variation by College 

The scholarship program at BMCC and Hostos was programmatically identical, and 
thus findings have been pooled for the main outcomes of interest. In light of a desire to give 
feedback to the colleges, the impacts of the scholarship program were estimated separately as 
well. Table 3.3 presents the findings from this exploratory analysis of academic outcomes 
separately for BMCC and Hostos. 4 

As shown, the program’s impacts at the two colleges are very different from one anoth- 
er over the follow-up period. For example, at BMCC, program group students earned an 
average of 25.6 credits and control group students earned an average of 26.0 credits. This 
difference is not statistically significant. At Hostos, program group students earned an average 
of 30.7 credits and control group students earned an average of 24.4 credits. This difference of 
6.2 credits is statistically significant at the 1 percent level, and represents an increase of over 25 
percent. This impact accounts for about two full courses over the four-semester follow-up 
period. Finally, the difference in the impacts between the two colleges on this measure is also 
significant at the 1 percent level. 

Analyses were done to detennine what was driving this impact at Hostos. Were the in- 
creases all made while the scholarship was available, or did early impacts persist in the absence 
of the scholarship? Figure 3.2 details the impacts on registration over the four main semesters of 
follow-up at Hostos. Students in both groups registered at roughly the same rates in the first 
program semester, but they were already registered at the time of random assignment and thus 
no impacts on registration were expected. Looking at the later semesters, program group 
students registered at higher rates in the second semester of the scholarship as well as the third 
and fourth semesters when the scholarship offer was no longer available (8.0 percentage point 
impact in the second semester, 9.2 percentage point impact in the third semester, and 9.9 
percentage point impact in the fourth semester, all significant at the 10 percent level). There are 
also sizable impacts on full-time enrollment, credits attempted, and credits earned in all four 
semesters of follow-up. (See Appendix Table B.4.) These findings suggest that the impacts at 


4 Subgroup analysis by institution was not designated ahead of time. In addition, the study was not originally 
powered to look at institutional differences, and thus the analyses were not meant to be disaggregated by college. 
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The Performance-Based Scholarship Demonstration 
Table 3.3 

Cumulative Academic Outcomes Among Sample Members, by College Subgroup: 

First Through Fourth Semesters 


New York Final Report 




BMCC 



Hostos 


Difference 

Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Program 

Group 

Control 

Group 

Standard 
Difference Error 

Between 

Subgroups 

Number of semesters completed 

3.1 

3.1 

0.0 

0.1 

3.3 

2.9 

Q *** 

0.2 

tt 

Average number of credits attempted 

37.0 

36.7 

0.3 

1.1 

40.0 

33.7 

6.3 *** 

1.9 

ttt 

College-level credits 

26.4 

26.2 

0.2 

0.9 

31.8 

26.6 

5 9 *** 

1.7 

tt 

Developmental credits 

10.7 

10.5 

0.1 

0.5 

8.2 

7.1 

1.1 

0.8 


Average number of credits earned 

25.6 

26.0 

-0.4 

1.0 

30.7 

24.4 

6.2 *** 

1.9 

ttt 

College-level credits 

20.9 

21.0 

-0.1 

0.9 

25.6 

20.1 

5.6 *** 

1.7 

ttt 

Developmental credits 

4.7 

5.0 

-0.3 

0.3 

5.0 

4.3 

0.7 

0.6 


Cumulative GP A (%) 

3.0 to 4.0 

33.1 

35.7 

-2.7 

2.8 

29.3 

34.9 

-5.6 

5.1 


2.0 to 2.9 

36.1 

35.1 

1.1 

2.8 

49.1 

41.0 

8.1 

5.5 


Less than 2.0 

24.4 

22.2 

2.2 

2.5 

18.6 

20.5 

-1.9 

4.4 


No GPA a 

6.5 

7.1 

-0.6 

1.5 

3.0 

3.6 

-0.6 

2.0 


Sample size (total = 1,502 ) 

587 

582 



167 

166 





(continued) 






Table 3.3 (continued) 


SOURCE: MDRC calculations from Borough of Manhattan Community College (BMCC) and Hostos Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * 
= 10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance levels are indicated as: ttt = 1 percent; ft = 5 
percent; f = 1 0 percent. 

Estimates are adjusted by research cohort. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits and grades are not included in the outcomes shown 
in this table. 

A few college-level courses have excess "compensatory credits" associated with them. These credits are used to calculate total credit hours, but do not 
count toward a degree. In this table, compensatory credits are included in the developmental credits category and make up less than 1 percent of those 
credits. 

a The "No GPA" category includes students who are not enrolled and students who took only non-letter-graded courses, which are not included in GPA 



The Performance-Based Scholarship Demonstration 
Figure 3.2 

Enrollment During the First Four Semesters, Hostos Community College 

New York Final Report 



First semester Second semester Third semester Fourth semester 


■ Program □ C ontrol 

SOURCE: MDRC calculations from Hostos Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort. 

Students who dropped or withdrew from all of their courses are not considered enrolled. 
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Hostos were not concentrated in the first two semesters, when program group students were 
eligible for the scholarship, but persisted in the absence of the scholarship into the third and 
fourth semesters. Far from dissipating over time, the impacts on registration seem to grow from 
semester to semester. Appendix Table B.5 details the impacts by term for BMCC alone. As 
Appendix Table B.5 shows, no impacts were found on any academic measures at BMCC. 5 

What could be causing these potential differences in the program’s effectiveness at the 
two colleges? There are a few possible explanations. First, the colleges themselves are different. 
As discussed earlier, BMCC is a much larger institution, located in Manhattan. It serves sizable 
numbers of students commuting from Brooklyn (34.4 percent), Manhattan (24.6 percent), 
Queens (17.9 percent), and the Bronx (16.9 percent), whereas Hostos tends to serve students 
concentrated in the Bronx (63.4 percent) and, to a much smaller degree, Manhattan (14.4 
percent). 6 The institutional context may matter — a smaller campus where students may be 
more apt to get to know program administrators and seek out help when they need it may be 
more effective at operating this scholarship program than a larger campus. 

Second, the populations are different, as mentioned in the section on baseline character- 
istics. At Hostos, students tended to be older, more likely to have children, more likely to be 
Hispanic, and more likely to be receiving government benefits such as Temporary Assistance 
for Needy Families or food stamps. It is possible that the scholarship affected students at Hostos 
differently owing to some of these characteristics. 

Third, the program might have been implemented slightly differently. The site coordi- 
nator at Hostos was part of the student services division, where students could take advantage of 
services like academic advising, while at BMCC, the program was completely managed by the 
financial aid office. As mentioned in Chapter 3, the offices are set up differently and staffed by 
personnel who can provide different types of assistance to students. When students had ques- 


5 These findings appear robust to a number of sensitivity checks, such as dropping the largest 1 percent and 
largest 5 percent of observations. The three research groups were also examined by institution and it was found 
that impacts occurred only at Hostos and are largely concentrated in the PBS-plus-summer group. Further, 
these impacts remained despite trimming the sample of observations as described above. A comparison of the 
baseline characteristics of the research groups at Hostos revealed a few measures or characteristics that were 
different between the program and control groups, but no more than would be expected as a result of chance. 
An omnibus test was conducted to assess whether overall systematic differences in baseline characteristics at 
Hostos were observed between the program and control groups. The model’s likelihood ratio test yielded a p- 
value of 0.55. Convention suggests that this probability is large enough that these potential differences can be 
ignored in the analyses. In addition, characteristics of the PBS and PBS-plus-summer groups at Hostos were 
compared. The omnibus test yielded a p-value of 0.01, suggesting that there are systematic differences between 
the two program groups at baseline. 

6 For BMCC, see www.bmcc.cuny.edu and www.bmcc.cuny.edu/iresearch/upload/Fall08FactSheet.pdf; 
for Hostos, see www.hostos.cuny.edu and www.hostos.cuny.edu/oaa/oir/PublicDocuments/StudentProfile.pdf. 
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tions about their awards, at BMCC they would receive financial assistance only, whereas at 
Hostos, students could access any number of other services. BMCC also had more than three 
times as many students to manage in the program. It is likely that the coordinator at Hostos 
became more acquainted with the students who came to the office with questions about their 
scholarship, simply because there were fewer of them to get to know. However, payments were 
not tied to receiving certain services or visiting the student services office, and students only 
visited the office in relation to the scholarship if they had a question about their award. (Pay- 
ments were mailed to students or transferred via direct deposit.) While there is suggestive 
evidence that students at Hostos received additional ad hoc advising, owing to the differences 
described above, this interaction was unplanned and unstructured. 

MDRC did not anticipate finding these differential impacts by college, and the quantita- 
tive and qualitative research was not geared to inform any of these theories in particular. In 
addition, as mentioned earlier, MDRC was unable to interview program staff; such interviews 
might have helped in examining some of these theories. Last, multiple comparison adjustments 
were not applied to these findings, as they are exploratory; exploratory analyses do not provide 
rigorous evidence of the program’s overall effectiveness . 7 Thus the differences in impacts 
between the colleges and the potential explanations offered for these differences should be an 
area for future research. 


7 Schochet (2008) suggests that multiplicity adjustments are not required for exploratory analyses. 
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Chapter 4 

Conclusions 


While the overall program in New York showed some initial promise in the program semesters, 
the long-term findings, pooled across both colleges, find no evidence that perfonnance-based 
scholarships led to an increase in the cumulative credits earned or number of semesters regis- 
tered. However, when the impacts are examined by college, there is exploratory evidence that 
the program worked better at one of the colleges. Here, the program had positive effects on 
registration, full-time enrollment, and credit accumulation, even in the absence of the scholar- 
ship offer. 

Across both colleges, there is evidence that offering the scholarship over the summer 
led to a modest impact on summer enrollment, but this did not translate into long-tenn academic 
gains. That is, while there were sizable impacts in the summer term where some portion of the 
program group was offered a scholarship, these impacts did not persist into later follow-up 
periods. Overall, the PBS-plus-summer group did not attempt or earn more credits over the two- 
year follow-up period than the PBS group or the control group. 

Finally, subgroup analyses of parents find that the scholarship had different impacts on 
parents compared with nonparents; however, these impacts are confounded with the differential 
impacts by college. At BMCC, there are no statistically significant impacts for parents or non- 
parents, while at Hostos, there are impacts for both parents and nonparents, but the difference 
between these two sets of impacts is not statistically significant. Parents were a particular 
subgroup of interest given the strong impacts found in the Opening Doors Louisiana program, 
which was targeted toward parents and had strong, positive impacts on academic success. 

Why Did the Program Work Differently at the Two Colleges? 

These results seem mixed — there are some positive findings, but in general, the impacts are 
not as strong as those seen in other studies of perfomiance-based scholarships in Louisiana, 
New Mexico, and Ohio. The data that were collected during scholarship disbursement monitor- 
ing suggest that the program, as intended, was implemented with fidelity to the design at both 
colleges — a scholarship-only program where students are paid in increments over the semes- 
ter, based on achieving certain academic benchmarks. The natural questions are why didn’t this 
program work better, especially at BMCC, and, how is the study at BMCC different from the 
study at Hostos? 

Some possible reasons were discussed in Chapter 3. First, the colleges themselves are 
different — BMCC is a larger institution, serving individuals from all around the New York 
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City area, while Hostos is much smaller, with students hailing predominantly from the Bronx. 
Second, the populations are different — Hostos students in the sample tended to be older, more 
likely to have children, more likely to be Hispanic, and more likely to be receiving government 
benefits. And, third, the programs might have been implemented differently in the two colleges, 
owing to the housing of the program in the student services division at Hostos versus the 
financial aid office at BMCC. 

Another possible reason is that the theory of change behind the program design was 
simply incorrect: a bare-bones, scholarship-only program may not be enough to create incen- 
tives for making academic progress among low-income students, and Hostos is an outlier in its 
success. The data collected during one-on-one interviews highlighted some of the struggles that 
students in the sample faced, but program group students in general did not think that the 
scholarship was meaningful in terms of reported behavioral change. Furthermore, these impres- 
sions were similar across both colleges. It is possible that the scholarship award was not salient 
enough to students. Given the high cost of living in New York City, perhaps $1,300 was too 
low an amount to provide strong incentives for students. Since students at Hostos were more 
likely to be receiving government benefits, a proxy for low-income status, it is thus conceivable 
that the award meant something different or something more to students at Hostos, accounting 
for some of the difference in impacts. 

These are all theories, and the main cause of this difference in impacts is unknown. The 
cause could be related to any of these factors, or others that were not considered, and the extent 
to which these theories explain the results could also vary. MDRC’s qualitative research was 
not geared to answer many of these questions, and the implementation data that were collected 
were limited in scope. Fortunately, the New York program is just one test of this design in the 
broader PBS Demonstration. The studies in Ohio and California also test a scholarship-only 
program. While each of these sites targets a different population, their collective results can help 
to answer this important question in the coming years. 

Important Lessons Learned 

While the longer-term pooled findings suggest that the program did not help (nor harm) 
students, there are important lessons that can be gathered from the results of the study. 

First, the summer scholarship had sizable impacts on registration in the summer semes- 
ter. Though these effects did not persist, it is possible that offering students additional money in 
the summer semesters over multiple summers would lead to more meaningful impacts. Cou- 
pling this with student services to advise students that attending school in the summer may help 
toward earning their degree or certificate could be even more powerful. This view was the 
impetus behind the year-round Pell Grants, a federal program enacted in the 2009-2010 aca- 
demic year. The program allowed students to use Pell Grants to pay for summer courses, 
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helping students (often nontraditional students such as the sample in New York) to earn their 
degrees more quickly. 1 However, due to budget cuts and a desire to keep the maximum Pell 
Grant at $5,500, the program ceased to exist for the 2011-2012 academic year. 2 Perhaps 
offering the perfomiance-based scholarships in only the summer semesters could have yielded 
different results — an important area for future research. 

Second, students in the program in New York City received their checks in the mail or 
via direct deposit. Perhaps the delivery of the award was misguided, and the program would 
have worked better if there were a more personal component attached to the award, such as 
picking a check up from an advisor. In the Opening Doors study in Louisiana, an advising 
component was attached to the scholarship, and the hope was that counselors would get to know 
students on a more personal level, encourage them to persist in their efforts, and take an active 
role in referring students to services on campus and in the community if required. In practice, 
the counselors played more of a monitoring function — they checked on students’ academic 
benchmarks and met with students to explain the program rules and give them their checks. 
While the effects of the scholarship cannot be disentangled from the effects of the counseling, 
survey results from the study showed that program group students were more likely than those 
in the control group to say that they had a college contact for advice and support on personal 
matters. They were also more likely to say that they had a mentor at college for academic and 
career-related matters. It may be the case that performance-based scholarships are most effec- 
tive when paired with additional student supports. 

What’s Next? 

The scholarship program in the two colleges in New York City has continued after the study as 
part of a program of “sustainability scholarships” (that is, perfonnance-based scholarships that 
operate after the study is over) for a small number of students. These sustainability scholarships 
were established to ensure that the infrastructure to administer these scholarships remains in 
place if continued funding is warranted by the findings. Thus, both colleges continue to have 
students eligible for performance-based scholarships. At BMCC, MDRC is also working with 
the college to administer an additional scholarship opportunity for low-income men of color, 
with additional support services provided by the college as part of the scholarship opportunity. 

The study in New York is one of six studies being conducted as part of MDRC’s Per- 
fonnance-Based Scholarship Demonstration. It is but one piece of the financial-aid puzzle that 
the PBS Demonstration seeks to explain. MDRC will be publishing longer-range findings from 


'Quizon (2011). 
2 Nelson (2011). 
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Ohio and New Mexico, as well as forthcoming results from three more states in the demonstra- 
tion (California, Arizona, and Florida). These reports will answer pressing questions such as: 


• How does variation in the amount and duration of scholarships affect 
academic outcomes? The program in California incorporates a six-way de- 
sign, testing different amounts and durations in order to leam how impacts 
vary across these different spectrums. 3 An interim report planned for release 
in 2013 will shed light on this important question. 

• For which types of students do the scholarships work best? Examining 
academic outcomes across the six programs in the PBS Demonstration, in 
conjunction with the original findings from Opening Doors Louisiana, will 
provide telling infomiation on the types of students for whom the perfor- 
mance-based scholarship seems to work best. This can help answer questions 
such as: Do perfonnance-based scholarships work for students who have 
children? Traditional students? Developmental education students? Do per- 
fonnance-based scholarships work for various subgroups of students such as 
males or Hispanic students? 

• How does the addition of student services, such as advising and tutoring, 
affect the success of the program? Are scholarship dollars alone enough to 
create incentives for change in the populations of study? This is one of the 
lingering questions coming out of this report on the New York findings — is 
it possible that a scholarship-only program can produce meaningful impacts 
for low-income students, or are services required? 

MDRC plans to synthesize the results across these sites as time goes on. Collectively, 
these findings will add to the body of knowledge on the effectiveness of these scholarships on 
improving academic success for low-income students. 


3 Ware and Patel (2012). 
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The Performance-Based Scholarship Demonstration 
Appendix Table A.l 

Selected Characteristics of Sample Members at Baseline, by Research Group 

New York Final Report 


Characteristic 

Full 

Sample 

Program 

Group 

Control 

Group 

Gender (%) 

Male 

30.9 

30.2 

31.5 

Female 

69.1 

69.8 

68.5 

Age (%) 

22-26 years 

57.3 

57.3 

57.4 

27-30 years 

24.3 

24.7 

23.9 

3 1 years and over 

18.4 

18.0 

18.7 

Average age (years) 

26.5 

26.5 

26.6 

Marital status (%) 

Married 

18.2 

16.8 

19.6 

Unmarried 

74.0 

74.4 

73.5 

Missing 

7.8 

8.8 

6.8 

Race/ethnicity a (%) 

Hispanic 

44.3 

44.3 

44.3 

White 

6.1 

6.3 

5.8 

Black 

37.2 

36.2 

38.2 

Asian or Pacific Islander 

9.7 

10.3 

9.1 

Other 

2.7 

2.9 

2.6 

Number of children (%) 

0 

52.3 

53.9 

50.7 

1 

26.5 

24.4 

28.6 * 

2 

14.4 

15.2 

13.5 

3 or more 

6.8 

6.4 

7.3 

Among sample members with children 

Average age of youngest child (years) 

4.3 

4.4 

4.2 

Household receiving any government benefits (%) 

37.9 

37.3 

38.5 

Missing 

12.0 

11.7 

12.3 

Financially dependent on parents (%) 

1.3 

0.9 

1.6 

Missing 

4.4 

3.8 

5.0 

Currently employed (%) 

56.0 

56.5 

55.4 


(continued) 


51 






Appendix Table A.l (continued) 


Characteristic 

Full 

Sample 

Program 

Group 

Control 

Group 

Among those currently employed 
Number of hours worked per week in current job (%) 

1-10 hours 

6.4 

5.2 

7.7 

11-20 hours 

23.4 

26.0 

20.7 

21-30 hours 

22.4 

23.5 

21.3 

31-40 hours 

44.9 

42.1 

47.6 

More than 40 hours 

3.0 

3.2 

2.7 

Average hourly wage at current job ($) 

11.0 

11.1 

11.0 

Highest grade completed (%) 

1 Oth grade or lower 

16.0 

15.8 

16.2 

1 1 th grade 

12.8 

11.9 

13.6 

12th grade 

65.0 

66.2 

63.9 

Missing 

6.2 

6.1 

6.3 

Diplomas/degrees earned b (%) 

High school diploma 

65.0 

65.5 

64.5 

General Educational Development (GED) certificate 

33.1 

32.7 

33.6 

Occupational/technical certificate 

13.3 

11.8 

14.9 * 

Associate's degree or higher 

2.6 

2.4 

2.7 

None of the above 

2.4 

2.2 

2.6 

Date of high school graduation/ GED receipt (%) 

During the past year 

5.2 

4.0 

6.4 ** 

Between 1 and 5 years ago 

29.2 

31.0 

27.4 

Between 5 and 10 years ago 

37.5 

37.4 

37.7 

More than 1 0 years ago 

19.6 

19.5 

19.6 

Missing 

8.5 

8.1 

8.8 

Main reason for enrolling in college 13 (%) 

To complete a certificate program 

2.9 

3.0 

2.8 

To obtain an associate's degree 

50.7 

48.6 

52.9 * 

To transfer to a 4-year college/university 

43.1 

46.1 

40.0 ** 

To obtain/update job skills 

3.2 

2.8 

3.7 

Other 

1.6 

1.2 

2.0 

First person in family to attend college (%) 

32.9 

34.5 

31.4 

Language other than English spoken regularly in home (%) 

54.6 

54.4 

54.8 

Sample size 

1,502 

754 

748 
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Appendix Table A.l (continued) 


SOURCE: MDRC calculations using Baseline Information Form (B1F) data. 

NOTES: To analyze whether baseline characteristics jointly predicted research group status, a likelihood 
ratio test was performed. This yielded a p-value of 0.47. Convention suggests that these probabilities are 
large enough that these potential differences can be ignored in the analyses. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and college. 

Characteristics shown in italics indicate nonexperimental data. Significance tests are not calculated 
for nonexperimental data. 

Missing values are only included in variable distributions for characteristics with more than 5 percent 
of the sample missing. 

Distributions may not add to 100 percent because of rounding. 

Respondents who said they are Flispanic and chose a race are included only in the Hispanic 
category. Respondents who are not Hispanic and chose more than one race are considered multiracial. 
"Other" includes American Indian /Alaskan Native, multiracial, and other. 

b Distributions may not add to 100 percent because categories are not mutually exclusive. 
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Appendix B 

Additional Impact Tables 




The Performance-Based Scholarship Demonstration 
Appendix Table B.l 

Academic Outcomes Among Sample Members: First Through Fourth Semesters 


New York Final Report 



Program 

Control 


Standard 

Outcome 

Group 

Group 

Difference 

Error 


First semester 


Enrolled as of semester end (%) 

95.6 

94.8 

0.8 

1.1 

Full time 3 

77.9 

73.8 

4.1 * 

2.2 

Part time b 

17.4 

20.6 

-3.2 

2.0 

Average number of credits attempted 

12.6 

12.2 

0.4 ** 

0.2 

College-level credits 

7.7 

7.5 

0.3 

0.2 

Developmental credits 

4.9 

4.8 

0.2 

0.2 

Average number of credits earned 

8.9 

8.4 

0.5 ** 

0.3 

College-level credits 

6.2 

5.8 

0.4 * 

0.2 

Developmental credits 

2.7 

2.5 

0.1 

0.2 

Earned a "C" or better in 6 or more credits (%) 

71.4 

70.5 

0.9 

2.3 

Term GPA (%) 

3.0 to 4.0 

40.1 

39.0 

1.0 

2.5 

2.0 to 2.9 

29.4 

27.0 

2.4 

2.3 

Less than 2.0 

19.0 

20.2 

-1.2 

2.0 

No GPA C 

11.5 

13.8 

-2.2 

1.7 

Second semester 

Enrolled as of semester end (%) 

78.1 

76.6 

1.6 

2.1 

Full time 3 

60.6 

54.7 

6.0 ** 

2.5 

Part time b 

16.5 

20.3 

-3.9 * 

2.0 

Average number of credits attempted 

9.9 

9.5 

0.4 

0.3 

College-level credits 

7.5 

7.0 

0.5 * 

0.3 

Developmental credits 

2.5 

2.5 

0.0 

0.2 

Average number of credits earned 

7.0 

6.8 

0.2 

0.3 

College-level credits 

5.9 

5.6 

0.3 

0.3 

Developmental credits 

1.1 

1.2 

-0.1 

0.1 

Earned a "C" or better in 6 or more credits (%) 

57.4 

56.8 

0.6 

2.5 

Term GPA(%) 

3.0 to 4.0 

32.2 

34.5 

-2.3 

2.4 

2.0 to 2.9 

26.5 

22.6 

4.0 * 

2.2 

Less than 2.0 

16.5 

15.6 

0.8 

1.9 

No GPA C 

24.8 

27.3 

-2.5 

2.3 
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Appendix Table B.l (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Third semester 

Enrolled as of semester end (%) 

61.9 

60.7 

1.2 

2.5 

Full time a 

43.9 

41.3 

2.6 

2.6 

Part time b 

15.2 

17.1 

-1.9 

1.9 

Average number of credits attempted 

7.5 

7.3 

0.3 

0.3 

College-level credits 

6.1 

5.9 

0.2 

0.3 

Developmental credits 

1.4 

1.3 

0.1 

0.1 

Average number of credits earned 

5.5 

5.2 

0.3 

0.3 

College-level credits 

5.0 

4.7 

0.3 

0.3 

Developmental credits 

0.5 

0.5 

0.0 

0.1 

Earned a "C" or better in 6 or more credits (%) 

44.0 

44.2 

-0.2 

2.6 

Term GPA(%) 

3.0 to 4.0 

27.1 

25.9 

1.2 

2.3 

2.0 to 2.9 

20.7 

19.9 

0.8 

2.1 

Less than 2.0 

12.2 

12.4 

-0.3 

1.7 

No GPA C 

40.1 

41.7 

-1.7 

2.5 

Fourth semester 

Enrolled as of semester end (%) 

51.2 

49.5 

1.8 

2.6 

Full time 3 

34.0 

29.9 

4.0 * 

2.4 

Part time b 

14.6 

16.7 

-2.1 

1.9 

Average number of credits attempted 

6.0 

5.6 

0.4 

0.3 

College-level credits 

5.0 

4.7 

0.3 

0.3 

Developmental credits 

1.0 

0.9 

0.1 

0.1 

Average number of credits earned 

4.1 

4.1 

0.0 

0.3 

College-level credits 

3.8 

3.7 

0.1 

0.3 

Developmental credits 

0.3 

0.4 

-0.1 

0.1 

Earned a "C" or better in 6 or more credits (%) 

33.4 

34.1 

-0.7 

2.4 

Term GPA (%) 

3.0 to 4.0 

23.2 

20.8 

2.4 

2.1 

2.0 to 2.9 

13.7 

17.0 

-3.3 * 

1.9 

Less than 2.0 

12.6 

9.8 

2.8 * 

1.6 

No GPA C 

50.5 

52.4 

-1.9 

2.6 

Sample size (total = 1,502 ) 

754 

748 
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Appendix Table B.l (continued) 


SOURCE: MDRC calculations from Borough of Manhattan Community College and Hostos Community 
College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and college. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits and 
grades are not included in the outcomes shown in this table. 

Outcomes in individual semesters include fall and spring semesters only. Fall semester outcomes include any 
winter sessions. Outcomes for the summer sessions are shown separately, in Table 3.2. 

A few college-level courses have excess "compensatory credits" associated with them. These credits are used 
to calculate total credit hours, but do not count toward a degree. In this table, compensatory credits are included 
in the developmental credits category and make up less than 1 percent of those credits. 

a Full-time enrollment is defined as 12 or more credits. 

b Part-time enrollment is defined as 6 credits to fewer than 12 credits. 

c The "No GPA" category includes students who are not enrolled and students who took only non-letter- 
graded courses, which are not included in GPA calculations. 
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The Performance-Based Scholarship Demonstration 
Appendix Table B.2 

Credit Accumulation During the First Four Semesters 


New York Final Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Cumulative credits earned as of end of 
First semester 

8.9 

8.4 

0.5 ** 

0.3 

Second semester 

16.3 

15.5 

0.9 * 

0.5 

Third semester 

22.4 

21.3 

1.1 

0.7 

Fourth semester 

26.7 

25.6 

1.1 

0.9 

Sample size (total = 1,502 ) 

754 

748 




SOURCE: MDRC calculations from Borough of Manhattan Community College and Hostos 
Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between the research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and college. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their 
credits are not included in the outcomes shown in this table. 
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The Performance-Based Scholarship Demonstration 
Appendix Table B.3 

Cumulative Academic Outcomes Among Sample Members, by Parental Status: 

First Through Fourth Semesters 


New York Final Report 




Parent 



Not Parent 


Difference 

Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Between 

Subgroups 

Number of semesters completed 

3.1 

3.0 

0.1 

0.1 

3.2 

3.1 

0.0 

0.1 


Average number of credits attempted 

37.5 

34.3 

3.2 

** 1.3 

37.9 

37.8 

0.1 

1.3 


College-level credits 

27.2 

24.9 

2.3 

** 1.2 

28.0 

27.6 

0.3 

1.2 


Developmental credits 

10.3 

9.5 

0.8 

0.7 

9.9 

10.1 

-0.2 

0.6 


Average number of credits earned 

26.5 

23.8 

2.7 

** 1.3 

26.9 

27.5 

-0.5 

1.2 

t 

College-level credits 

21.7 

19.3 

2.5 

** i i 

22.2 

22.3 

-0.1 

1.1 


Developmental credits 

4.8 

4.6 

0.2 

0.4 

4.7 

5.1 

-0.5 

0.4 


Cumulative GPA (%) 

3.0 to 4.0 

31.0 

33.4 

-2.4 

3.5 

33.4 

38.0 

-4.6 

3.4 


2.0 to 2.9 

40.3 

36.7 

3.6 

3.7 

37.7 

36.4 

1.3 

3.5 


Less than 2.0 

22.0 

23.9 

-1.9 

3.2 

24.0 

18.9 

5.1 * 

2.9 


No GPA a 

6.7 

6.0 

0.7 

1.8 

4.9 

6.7 

-1.8 

1.7 


Sample size (total = 1 ,493 ) 

347 

366 



404 

376 
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SOURCE: MDRC calculations from Borough of Manhattan Community College and Hostos Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance levels are indicated as: f f f = 1 percent; f f = 5 percent; 
f = 1 0 percent. 

Estimates are adjusted by research cohort and college. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits and grades are not included in the outcomes shown in 
this table. 

A few college-level courses have excess "compensatory credits" associated with them. These credits are used to calculate total credit hours, but do not count 
toward a degree. In this table, compensatory credits are included in the developmental credits category and make up less than 1 percent of those credits. 

“The "No GPA" category includes students who are not enrolled and students who took only non-letter-graded courses, which are not included in GPA 
calculations. 



The Performance-Based Scholarship Demonstration 
Appendix Table B.4 


Academic Outcomes Among Sample Members at Hostos Community College: 

First Through Fourth Semesters 

New York Final Report 



Program 

Control 


Standard 

Outcome 

Group 

Group 

Difference 

Error 


First semester 


Enrolled as of semester end (%) 

96.4 

94.0 

2.4 

2.3 

Full time a 

80.3 

71.1 

9.2 * 

4.7 

Part time b 

16.1 

22.3 

-6.2 

4.3 

Average number of credits attempted 

13.3 

12.3 

1.0 ** 

0.5 

College-level credits 

OO 

OO 

8.5 

0.3 

0.4 

Developmental credits 

4.5 

3.8 

0.7 * 

0.4 

Average number of credits earned 

10.2 

9.0 

1.2 * 

0.6 

College-level credits 

7.0 

6.4 

0.6 

0.4 

Developmental credits 

3.1 

2.6 

0.5 

0.4 

Earned a "C" or better in 6 credits or more (%) 

81.4 

72.9 

8.5 * 

4.6 

Term GPA(%) 

3.0 to 4.0 

41.9 

37.9 

4.0 

5.4 

2.0 to 2.9 

35.3 

32.6 

2.7 

5.2 

Less than 2.0 

18.6 

20.5 

-1.9 

4.4 

No GPA C 

4.2 

9.0 

-4.8 * 

2.7 

Second semester 

Enrolled as of semester end (%) 

81.5 

73.5 

* 

O 

OO 

4.6 

Full time a 

67.7 

53.0 

14 7 *** 

5.3 

Part time b 

13.2 

18.7 

-5.5 

4.0 

Average number of credits attempted 

10.7 

9.2 

15** 

0.7 

College-level credits 

8.7 

7.3 

13 ** 

0.6 

Developmental credits 

2.0 

1.9 

0.2 

0.3 

Average number of credits earned 

8.4 

6.7 

1 7 *** 

0.6 

College-level credits 

7.1 

5.6 

1 5 *** 

0.5 

Developmental credits 

1.3 

1.1 

0.2 

0.3 

Earned a "C" or better in 6 credits or more (%) 

68.3 

58.4 

9.9 * 

5.3 

Term GPA(%) 

3.0 to 4.0 

31.7 

27.7 

4.0 

5.0 

2.0 to 2.9 

34.8 

29.5 

5.3 

5.1 

Less than 2.0 

14.4 

15.1 

-0.7 

3.9 

No GPA C 

19.1 

27.7 

-8.6 * 

4.6 
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Appendix Table B.4 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Third semester 

Enrolled as of semester end (%) 

67.7 

58.4 

9.2 * 

5.3 

Full time a 

47.3 

36.1 

1 1 2 ** 

5.4 

Part time b 

17.3 

19.9 

- 2.6 

4.3 

Average number of credits attempted 

8.2 

6.7 

1 5 ** 

0.7 

College-level credits 

7.2 

5.8 

1.3 ** 

0.6 

Developmental credits 

1.0 

0.9 

0.1 

0.2 

Average number of credits earned 

6.1 

4.6 

1 5 ** 

0.6 

College-level credits 

5.8 

4.3 

[ 5 ** 

0.6 

Developmental credits 

0.3 

0.3 

0.0 

0.1 

Earned a "C" or better in 6 credits or more (%) 

51.5 

42.7 

8.8 

5.4 

Term GPA(%) 

3.0 to 4.0 

30.0 

21.7 

8.3 * 

4.8 

2.0 to 2.9 

25.7 

24.7 

1.0 

4.8 

Less than 2.0 

11.3 

12.1 

-0.7 

3.5 

No GPA C 

32.9 

41.6 

- 8.6 

5.3 

Fourth semester 

Enrolled as of semester end (%) 

56.3 

46.4 

9.9 * 

5.5 

Full time 3 

36.0 

19.8 

16.1 *** 

4.8 

Part time b 

18.6 

19.9 

-1.3 

4.3 

Average number of credits attempted 

6.5 

4.7 

1 g *** 

0.7 

College-level credits 

5.9 

4.2 

1 g *** 

0.6 

Developmental credits 

0.5 

0.5 

0.1 

0.2 

Average number of credits earned 

4.8 

3.5 

1.3 ** 

0.6 

College-level credits 

4.7 

3.2 


0.5 

Developmental credits 

0.1 

0.3 

- 0.1 

0.1 

Earned a "C" or better in 6 credits or more (%) 

40.7 

28.9 

11.8 ** 

5.2 

Term GPA (%) 

3.0 to 4.0 

28.8 

18.7 

10.1 ** 

4.6 

2.0 to 2.9 

15.6 

18.7 

-3.1 

4.1 

Less than 2.0 

12.0 

8.4 

3.5 

3.3 

No GPA C 

43.7 

54.2 

-10.5 * 

5.5 

Sample size (total = 333 ) 

167 

166 
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Appendix Table B.4 (continued) 


SOURCE: MDRC calculations from Hostos Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits and 
grades are not included in the outcomes shown in this table. 

Outcomes in individual semesters include fall and spring semesters only. Fall semester outcomes include any 
winter sessions. 

A few college-level courses have excess "compensatory credits" associated with them. These credits are used 
to calculate total credit hours, but do not count toward a degree. In this table, compensatory credits are included 
in the developmental credits category and make up less than 5 percent of those credits. 

a Full-time enrollment is defined as 12 credits or more. 

b Part-time enrollment is defined as 6 credits to fewer than 12 credits. 

c The "No GPA" category includes students who are not enrolled and students who took only non-letter- 
graded courses, which are not included in GPA calculations. 
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The Performance-Based Scholarship Demonstration 
Appendix Table B.5 


Academic Outcomes Among Sample Members at 
Borough of Manhattan Community College: First Through Fourth Semesters 

New York Final Report 



Program 

Control 


Standard 

Outcome 

Group 

Group 

Difference 

Error 


First semester 


Enrolled as of semester end (%) 

95.4 

95.0 

0.4 

1.3 

Full time a 

77.2 

74.6 

2.6 

2.5 

Part time b 

17.7 

20.1 

-2.4 

2.3 

Average number of credits attempted 

12.5 

12.2 

0.3 

0.2 

College-level credits 

7.4 

7.2 

0.3 

0.2 

Developmental credits 

5.0 

5.0 

0.0 

0.2 

Average number of credits earned 

8.5 

8.2 

0.3 

0.3 

College-level credits 

6.0 

5.7 

0.3 

0.2 

Developmental credits 

2.5 

2.5 

0.0 

0.2 

Earned a "C" or better in 6 credits or more (%) 

68.5 

69.8 

-1.3 

2.7 

Term GPA(%) 

3.0 to 4.0 

39.5 

39.3 

0.2 

2.9 

2.0 to 2.9 

27.8 

25.4 

2.3 

2.6 

Less than 2.0 

19.1 

20.1 

-1.0 

2.3 

No GPA C 

13.6 

15.1 

-1.5 

2.1 

Second semester 

Enrolled as of semester end (%) 

77.2 

77.5 

-0.3 

2.4 

Full time a 

58.6 

55.1 

3.5 

2.9 

Part time b 

17.4 

20.8 

-3.4 

2.3 

Average number of credits attempted 

9.7 

9.6 

0.1 

0.3 

College-level credits 

7.1 

6.9 

0.2 

0.3 

Developmental credits 

2.6 

2.7 

-0.1 

0.2 

Average number of credits earned 

6.6 

6.8 

-0.2 

0.3 

College-level credits 

5.5 

5.6 

-0.1 

0.3 

Developmental credits 

1.1 

1.2 

-0.1 

0.1 

Earned a "C" or better in 6 credits or more (%) 

54.4 

56.3 

-2.0 

2.9 

Term GPA(%) 

3.0 to 4.0 

32.4 

36.4 

-4.0 

2.8 

2.0 to 2.9 

24.2 

20.6 

3.6 

2.4 

Less than 2.0 

17.0 

15.8 

1.2 

2.2 

No GPA C 

26.4 

27.2 

-0.8 

2.6 
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Appendix Table B.5 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Third semester 

Enrolled as of semester end (%) 

60.3 

61.3 

-1.0 

2.9 

Full time a 

42.9 

42.8 

0.1 

2.9 

Part time b 

14.7 

16.3 

-1.7 

2.1 

Average number of credits attempted 

7.4 

7.4 

-0.1 

0.4 

College-level credits 

5.8 

6.0 

-0.1 

0.3 

Developmental credits 

1.5 

1.5 

0.0 

0.2 

Average number of credits earned 

5.3 

5.4 

-0.1 

0.3 

College-level credits 

4.8 

4.8 

0.0 

0.3 

Developmental credits 

0.5 

0.6 

-0.1 

0.1 

Earned a "C" or better in 6 credits or more (%) 

41.9 

44.7 

-2.8 

2.9 

Term GPA(%) 

3.0 to 4.0 

26.2 

27.1 

-0.9 

2.6 

2.0 to 2.9 

19.2 

18.6 

0.7 

2.3 

Less than 2.0 

12.4 

12.6 

-0.1 

1.9 

No GPA C 

42.1 

41.7 

0.3 

2.9 

Fourth semester 

Enrolled as of semester end (%) 

49.8 

50.3 

-0.6 

2.9 

Full time 3 

33.4 

32.8 

0.6 

2.8 

Part time b 

13.5 

15.8 

-2.3 

2.1 

Average number of credits attempted 

5.9 

5.9 

0.0 

0.4 

College-level credits 

4.8 

4.9 

-0.1 

0.3 

Developmental credits 

1.1 

1.0 

0.1 

0.1 

Average number of credits earned 

3.9 

4.3 

-0.4 

0.3 

College-level credits 

3.6 

3.8 

-0.3 

0.3 

Developmental credits 

0.4 

0.4 

-0.1 

0.1 

Earned a "C" or better in 6 credits or more (%) 

31.3 

35.6 

-4.2 

2.8 

Term GPA (%) 

3.0 to 4.0 

21.6 

21.5 

0.2 

2.4 

2.0 to 2.9 

13.1 

16.5 

-3.4 

2.1 

Less than 2.0 

12.8 

10.1 

2.6 

1.9 

No GPA C 

52.5 

51.9 

0.6 

2.9 

Sample size (total = 1,169 ) 

587 

582 
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Appendix Table B.5 (continued) 


SOURCE: MDRC calculations from Borough of Manhattan Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort. 

Students who dropped or withdrew from all of their courses are not considered enrolled. Their credits and 
grades are not included in the outcomes shown in this table. 

Outcomes in individual semesters include fall and spring semesters only. Fall semester outcomes include any 
winter sessions. 

a Full-time enrollment is defined as 12 credits or more. 
b Part-time enrollment is defined as 6 credits to fewer than 12 credits. 

°The "No GPA" category includes students who are not enrolled and students who took only non-letter- 
graded courses, which are not included in GPA calculations. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



